PCT 



WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification 6 : 
G06T 11/20 



Al 



(11) International Publication Number: 
(43) International Publication Date: 



WO 98/20458 

14 May 1998 (14.05.98) 



(21) International Application Number: PCT/US97/20394 

(22) International Filing Date: 7 November 1997 (07.1 1.97) 



(30) Priority Data: 

08/745,574 



8 November 1996 (08.1 1.96) US 



(71) Applicant: MIRROR SOFTWARE CORPORATION [US/US]; 

12530 - 128th Lane N.E., Kiricland. WA 98034 (US). 

(72) Inventors: LINDFORD. Ray, A.; 14128 - 126th Place N.E., 

Kiricland, WA 98034 (US). BLANCHARD, Perin; 22519 
N.E. 165th Court, Woodinville, WA 98072 (US). 

(74) Agent: TULLETT, Rodney, C; Christensen O'Connor Johnson 
& Kindness PLLC, Suite 2800, 1420 Fifth Avenue, Seattle, 
WA 98101 (US). 



(81) Designated States: AL, AM, AT, AU, AZ, BA, BB, BG, BR. 
BY, CA, CH, CN, CU. CZ, DE, DK, EE, ES, Fl, GB, GE, 
GH, HU, IL, IS, JP, KE, KG, KP, KR, KZ, LC, LK, LR. 
LS, LT, LU, LV, MD. MG, MK, MN, MW, MX. NO. NZ, 
PL, PT, RO, RU, SD, SE, SG, SI, SK, SL. TJ, TM, TR» 
TT, UA, UG, UZ. VN, YU, ZW, ARIPO patent (GH, KE, 
LS, MW, SD, SZ, UG, ZW), Eurasian patent (AM, AZ, BY. 
KG, KZ, MD, RU, TJ, TM), European patent (AT, BE, CH, 
DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, NL, PT, 
SE), OAPI patent (BF, BJ. CF, CG, CI, CM, GA. GN, ML, 
MR, NE, SN, TD, TG). 



Published 

With international search report 

Before the expiration of the time limit for amending the 
claims and to be republished in the event of the receipt of 
amendments. 



(54) Title: AESTHETIC IMAGING SYSTEM 
(57) Abstract 

Disclosed is an aesthetic imaging system (20) for use in editing digital images. The aesthetic 
imaging system includes an imaging program (21) that runs on a personal computer (28) having an 
image capture board (30), a monitor (32), a video source (34) for providing digital images to be 
edited by the aesthetic imaging system, and a pen and tablet (38) for use in editing the images. The 
imaging program includes a unique combination draw tool that includes a freehand draw mode, a 
curve mode and an undo mode that are available without cycling through menus. The combination 
draw tool may be used with any of the draw tools. Another feature of the imaging program is 
autoblend, a rectangular user interface that is invoked by each of the shape tools. The autoblend 
interface simplifies editing when using shape tools by consolidating the move, paste and blend, and 
paste without blending commands into a single, convenient interface. 
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AESTHETIC IMAGING SYSTEM 

Relationship to Other Applications 
This application is a continuation-in-part of U.S. patent application Serial 
No. 08/406,201, filed March 17, 1995, and U.S. patent application Serial 
5 No. 08/617,439, filed March 18, 1996, the benefit of the filing of which is hereby 
claimed under 35 U.S.C. § 120. 

Field of the Invention 
This invention generally relates to computer imaging programs and, more 
specifically, to a method and apparatus for manipulating digital photographs. 
10 Background of the Invention 

The digital age continues to present additional opportunities for visual 
communication using computers. As an example, digital photographs are routinely 
being manipulated to produce a desired effect or result in the magazine and film- 
making industries. In the medical field, computer-based imaging has and continues to 
15 gain acceptance in a clinical setting as a viable communications tool between plastic or 
"cosmetic" surgeons and potential patients. 

People are with increasing frequency consulting physicians about cosmetic 
surgery. While in many cases the patients considering cosmetic surgery have an 
impressive understanding of the procedures available and medical terms used to 
20 describe these procedures, it is apparent that the slightest miscommunication may 
result in dire consequences. This has promoted the use of computer imaging to 
facilitate communication between the physician and prospective patient. Specifically, 
high-end aesthetic imaging systems allow a physician to take pre-operative digital 
images of the patient, e.g., including profile and frontal views. The images are stored 
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in memory in the computer where they can then be edited. Using feedback from the 
patient, the edited images are useful in fully understanding the procedures desired. 
The visual support provided by a computer-based imaging system is extremely 
valuable on both sides. A cosmetic surgeon can more readily understand what 
5 patients hope to achieve by a cosmetic surgical procedure; and patients can view a 
detailed visual representation of predicted results, including both the benefits and 
limitations of the procedure. 

Existing aesthetic imaging systems typically use a number of tools to allow a 
physician to manipulate a pre-operative image of a patient to illustrate an intended 

10 post-operative result. Preferably, the tools allow the physician to manipulate the 
preoperative image during a consultation with a patient. By manipulating the image 
with the patient in attendance, the patient receives immediate feedback from the 
displayed results. For a successful preoperative consultation, the use of the editing 
tools should be as unobtrusive as possible. During the consultation, a physician would 

1 S like the patient to focus on the end results of the surgery, rather than the technologic 
wizardry used to demonstrate those results on the video monitor. Unfortunately, the 
editing tools used in existing aesthetic imaging systems typically hinder rather than 
help the physician in demonstrating the results that may be achieved through cosmetic 
surgery. 

20 Among others, a disadvantage of existing aesthetic imaging programs is that a 

physician or facilitator in a pre-operative consultation typically must go back and forth 
through many windows-based menus in order to edit an image. Cycling between the 
various menus to invoke the tools necessary for a consultation is disadvantageous in 
that it is time consuming. For example, some physicians schedule a follow-up visits 

25 for patients to allow the physician time to edit the images. More important, however, 
is that the process is distracting to the patient and tends to make the pre-operative 
consultation all the more mystifying. As a result of the disadvantages associated with 
prior art systems, some patients lose interest or become frustrated with the interview, 
both of which may reflect back on the physician. 

30 A further disadvantage of existing aesthetic imaging systems is that it is 

impossible for a physician or facilitator to display different combinations of the edits 
that they have performed. In existing aesthetic imaging programs, as a physician edits 
a patient's image, the physician's edits are added to the preexisting edits of the image. 
Most programs are only capable of showing two version of the patient's image; the 

35 unedited, original version, and the final edited version incorporating all of the 
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physician's changes. It is therefore difficult for the physician to show various 
combinations of the edits that had been performed. For example, a physician may edit 
an image to remove wrinkles around a patient's eyes and to narrow the patient's nose. 
Existing aesthetic imaging programs only allowed the physician to simultaneously 

5 show all of these changes. I£ for example, the patient wished to view the changes to 
the eyes without the changes to the nose, it was difficult for the physician to easily 
separate out the sequence of edits that had been performed to adjust the patient's 
nose. An improved aesthetic imaging system in which a physician can more easily edit 
pre-operative images in response to a patient's suggestions and inquiries would be 

0 extremely advantageous. 

A still further disadvantage of existing aesthetic imaging systems is that the 
systems allow a physician to perform nearly flawless editing of a patient's image. 
Unfortunately, the edits performed by a physician on an aesthetic imaging system are 
often unobtainable results that cannot be achieved when actual surgery is performed. 

5 Unless the physician is especially skilled at using the aesthetic imaging system, it is 
difficult to show the patient achievable results, which typically fall within a range 
somewhere between the original patient image, and the optimum results as displayed 
by the edited image on the screen. It therefore would be advantageous to develop an 
aesthetic imaging system that allowed a physician to display more realistic results that 

0 are achievable through surgery. 

Summary of the Invention 
The invention is an aesthetic imaging system for use in editing digital images. 
The aesthetic imaging system includes a unique user interface that allows edits to be 
performed more efficiently and with less confusion to the patient. 

S In one aspect of the invention, a method of editing a digital image comprised 

of a plurality of color pixels in an aesthetic imaging system is disclosed. The aesthetic 
imaging system including a processor, a memory, a monitor, and a pen and 
cooperating tablet for controlling a cursor displayed on the monitor. The pen has a 
depressable tip and a side button, each of which include an on status and an off status, 

0 wherein the position of the pen tip relative to the tablet determines the position of the 
cursor on the monitor. The method comprises: (a) evaluating the following variables: 
(i) the status of the tip of the pen; (ii) the status of the side button on the pen; 
and (iii) movement of the pen tip relative to the tablet; (b) actuating a freehand 
drawing mode if a first set of variables are present, wherein movement of the pen 

5 relative to the tablet edits pixels that are located at positions corresponding to the 
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position of the cursor; and (c) actuating a curve drawing mode if a second set of 
variables are present, wherein a line segment is displayed between two endpoints and 
movement of the pen relative to the tablet stretches the line segment, forming a curve 
and editing pixels that are located at positions corresponding to the position of the 
5 curve. 

In another aspect of the invention, method further includes: (a) actuating a 
freehand undo mode if a third set of variables are present, wherein movement of the 
pen relative to the tablet restores pixels that are located at positions corresponding to 
the position of the cursor to their pre-edited color, and (b) actuating a curve undo 
10 mode if a fourth set of variables are present, wherein a line segment is displayed 
between two endpoints and movement of the pen relative to the tablet stretches the 
line segment, forming a curve and restoring pixels that are located at positions 
corresponding to the position of the curve to their pre-edited color. 

In other aspects of the invention, the freehand draw mode is actuated if the tip 
15 of the pen is depressed and pressure is maintained while the tip is moved a 
predetermined distance. In another aspect, the curve draw mode is actuated if the tip 
of the pen is depressed and released within a predetermined distance. 

In a further aspect of the invention, the curve draw mode is actuated by: 
(a) establishing a first endpoint at the position of the pen when the second set of 
variables are present; and (b) monitoring the status of the tip of the pen and 
establishing a second endpoint at the position of the pen if the tip is toggled from an 
off state to an on state. 

In another aspect of the invention, an improved prioritize feature is described 
for viewing an image. A user may identify several areas in a modified patient image 

25 containing edits that alter the image from the original image. As each area is identified 
by the user, an identifying tag is assigned to each of the areas. When desiring to show 
various combinations of the edits that have been performed on the image, the user 
may select the areas to display using the identifying tags. A user may therefore 
quickly cycle through various permutations of the procedures that have been edited 

30 for patient display. 

In still another aspect of the invention, an improved user interface is provided 
to nunimize the distraction of a patient as the patient is watching the image being 
edited. Preferably, a menu bar on the top of the display is removed during most 
editing, so that only the image of the patient is displayed. When the menu bar must be 
35 displayed, the bar itself is transparent to allow the patient to see the image through the 
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menu bar. Only the commands and the outline of the menu bar are presented in a 
contrasting color, minimizing the overall visual impression created by the menu bar 

In yet another aspect of the invention, a warp tool is described that allows a 
user to quickly and easily manipulate various features in an image. To use the warp 
tool, a user first encircles a portion of the image to be edited. Once the area has been 
selected, the user may tip the pen to designate a stretch point within the selected area 
As the user floats the pen over the tablet, the image is then stretched as if pulled from 
the stretch point. Areas of the image in the direction of stretch are compressed, and 
areas away from the direction of stretch are expanded. Areas surrounding the 
warping area are automatically adjusted to ensure that there is no discontinuities with 
the warping area. The manipulation of the image is performed in real-time, allowing a 
patient to see the warping as it is being performed by a user. 

In accordance with still another aspect of the invention, a number of modules 
are provided to allow the user to improve the quality of an image, to analyze an 
.mage or to prepare an image for meetings and presentations. A color correction 
module allows the color of an original image to be closely matched with the color ofa 
target image. An orientation correction module allows the size and orientation of an 
onginal ,mage to be closely matched with the size and orientation ofa target image 
A measurement module allows angles, distances, areas, and proportions to be 
measured and recorded on an image for presentation to a patient or colleagues And a 
label module allows structures in a patient's image to be linked to textual descriptions 
The modules greatly improve the ability to compare two images or maintain accurate 
records of achievable surgical results. 

In accordance with yet another aspect of the invention, a module is also 
provuled that simulates the effect of laser resurfacing treatment on a patient The 
module accurately portrays the image of the patient immediately following surgery 
the ,mage of the patient during the healing process, and the image of the patient whe^ 
the patient has fully healed. 

An advantage of the tools, features, and modules described herein are that they 
•mprove the overall experience of a patient during a preoperative visit with a 
phys.aan. The powerful tools in the aesthetic imaging system allow the physician to 
easdy maxupulate the patient's image in response to feedback provided by the patient 
The aesthetic imaging system interface also allows the patient to focus on the image 
bang manipulated, rather than on the aspects of the aesthetic imaging system that 
allow the man-pulation. The end result is an improved preoperative visit that provides 
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a more realistic impression of the results that a physician may achieve through 
surgery. 

Brief Description of the Drawings 
The foregoing aspects and many of the attendant advantages of this invention 
5 will become more readily appreciated as the same becomes better understood by 
reference to the following detailed description, when taken in conjunction with the 
accompanying drawings, wherein: 

FIGURE 1 is a block diagram of an aesthetic imaging system in accordance 
with the invention; 

10 FIGURE 2 is a block diagram illustrating various buffers used by the aesthetic 

imaging system to store and manipulate data; 

FIGURE 3 is a flow chart illustrating an exemplary routine by which digital 
images may be viewed and edited using the aesthetic imaging system; 

FIGURE 4A is a flow chart of an exemplary routine for photographing 
1 5 patients in accordance with the invention; 

FIGURE 4B is a flow chart of an exemplary routine for calculating a 
checksum value and comparing the calculated value to a previously stored value to 
determine if an image has been altered; 

FIGURE 5 is a flow diagram of an exemplary routine for implementing a 
20 combination tool for use with various drawing (draw) tools in accordance with the 
invention; 

FIGURE 6 is a pictorial representation of an image to be edited; 

FIGURES 7A-7E are pictorial representations of editing an image using a 
prior art imaging program; 
25 FIGURES 8A-8E are pictorial representations of using the aesthetic imaging 

system to accomplish the identical edits shown in FIGURES 7A-7E; 

FIGURE 9A is a flow chart of an exemplary routine of a contour tool in 
accordance with the invention; 

FIGURES 9B-9C are pictorial representations illustrating the function of the 
30 contour tool of FIGURE 9A; 

FIGURES 9D-9G are pictorial representations illustrating exemplary edits that 
may be accomplished using the contour tool; 

FIGURE 10 is a flow chart of an exemplary routine for implementing an 
autoblend tool in accordance with the invention; 
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FIGURE 1 1 is a pictorial representation of a user interface for implementing 
the autoblend tool of FIGURE 10; 

FIGURE 12A is a flow chart of an exemplary routine illustrating a cutout tool 
in accordance with the invention; 
5 FIGURE 12B is a flow diagram of an exemplary routine illustrating a rotate 

tool in accordance with the invention; 

FIGURE 13 is a flow chart of an exemplary routine for viewing images in 
accordance with the invention; 

FIGURES 14A-14D are pictorial representations illustrating the effects of a 
1 0 compare feature in accordance with the invention; 

FIGURES 15A-15C illustrate a split image option of viewing images in 
accordance with the invention; 

FIGURE 16 is a pictorial representation illustrating the use of a translucent 
image to allow a patient to accurately position themselves in order to capture a second 
1 5 image having the same location and orientation as an original stored image; 

FIGURE 17 is a pictorial representation illustrating a compare image wherein 
a presurgical image of a patient is compared side-by-side with a postsurgical image 
having the same location and orientation; 

FIGURES 18A-18C are pictorial representations illustrating the use of a warp 
20 shape tool to edit a patient's image; 

FIGURES 19A-19E are pictorial representations illustrating the function of 
the warp tool; 

FIGURE 20 is a pictorial representation illustrating the use of a transparent 
menu bar when viewing an image of a patient; 
25 FIGURE 21 is a flow chart of an exemplary routine for implementing a zoom 

viewing feature in accordance with the invention; 

FIGURES 22A-22D are flow charts of exemplary routines for implementing a 
number of modules that may be used to optimize an image for presentations or 
display; 

30 FIGURE 23 is a pictorial representation illustrating the use of a color 

correction module in accordance with the invention; 

FIGURE 24 is a pictorial representation illustrating the use of an orientation 
correction module in accordance with the invention; 

FIGURES 25A-25D are pictorial representations illustrating the use of a 
3 5 measurement module in accordance with the invention; 
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FIGURE 26 is a pictorial representation illustrating the use of a labeling 
module in accordance with the invention; and 

FIGURES 27A-27E are block diagrams and pictorial representations of the 
implementation and use of a laser resurfacing simulation in accordance with the 
5 invention. 

Detailed Description of the Preferred Embodiment 
An aesthetic imaging system 20 in accordance with the invention is illustrated 
in FIGURE 1. The aesthetic imaging system 20 includes an imaging program 21 that 
runs on a processing unit 22 controlled by an operating system 24. A memory 26 is 

1 0 connected to the processing unit and generally comprises, for example, random access 
memory (RAM), read only memory (ROM), and magnetic storage media such as a 
hard drive, floppy disk, or magnetic tape. The processing unit and memory are 
typically housed within a personal computer 28 which may be, for example, a 
Macintosh™, International Business Machines (IBM™) or IBM-compatible personal 

15 computer. When used with IBM and IBM-compatible personal computers, the 
operating system 24 may be DOS based or may incorporate a windowing environment 
such as Microsoft Windows™ or OS/2™. 

The aesthetic imaging system also includes an image capture board 30 that is 
coupled to the processing unit 22, a monitor 32, video source 34, and printer 36. The 

20 video source, monitor, and printer are coupled to the processing unit 22 through the 
image capture board 30. The video source may include one or more video cameras, a 
VCR, a scanner, or similar source for providing digital images to be edited by the 
aesthetic imaging system. The aesthetic imaging system further includes a pointing 
device, which is preferably a stylus (pen) and tablet 38, that is connected to the 

25 processing unit 22. In addition, the aesthetic imaging system may include a modem 40 
to provide on-line capabilities to users of the system, such as technical support and 
teleconferencing. 

The image capture board 30 has a plurality of buffers in high-speed memory, 
e.g., RAM that are used by the imaging program 21 to provide very fast response 
30 times to image edits. With reference to FIGURE 2, four buffers are illustrated for use 
in explaining the operation of the aesthetic imaging system. These include an original 
image buffer 50, a modified image buffer 52, a current image buffer 54, and a working 
buffer 56. Suitable image capture boards for use in the aesthetic imaging system 
include the Targa +64 and Targa 2000 boards, distributed by Truevision, Inc. of 
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Indianapolis, Indiana. The buffers are discussed in regard to a single pose only, such 
as a profile or front view of a person. 

The original image buffer 50 contains an unedited digital image, for example, a 
side profile picture of a potential patient. The modified image buffer 52 contains any 
5 edits made to a copy of the original image. The modified image buffer is updated 
during a save and after each session. The current image buffer 54 contains 
information identical to the modified image buffer upon beginning a session. 
Thereafter, edits made to the current image are saved in the working buffer 56 as an 
overlay to the current image. During a save, the contents of the current image 
10 buffer 54 are copied to the modified image buffer 52, and the working buffer 56 is 
cleared. 

Prior to discussing the aesthetic imaging system in further detail, a 
compendium of terms used in the application may be helpful: 

1 5 Image A digital photograph or picture of a patient. 

Stylus The "pen" that may be used to select menus, modify 

images, and carry out other commands in the 
program. The stylus controls the cursor, just as a 
mouse pointing device does on a personal computer. 



20 



25 



Tablet, or 

Pad The electronic notepad used in conjunction with a 

stylus. The pen must be held relatively close to the 
pad in order for the pen to communicate with the 
tablet. Unlike a mouse, the tablet follows an X / Y 
grid that matches the monitor, i.e., if the pen is 
positioned at the top left corner of the tablet, the 
cursor is displayed at the top left corner on the 
30 monitor. 



Floating 



Moving the pen to move the cursor, without actually 
touching the tablet. 
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Selecting Selecting (also referred to as "tipping" or "pressing") 
the tip of the pen briefly onto the tablet. This selects 
a command or affects the drawing tool, depending on 
the current procedure being implemented. 

Cancel Using the side button on the pen to execute a 

command. 



10 



Moving Pressing the tip of the pen on the tablet, releasing it, 

then moving it across the tablet. 



Pressing or 
Tipping & 

Dragging Pressing the tip of the pen, then dragging it across the 
15 tablet while maintaining pressure. When using 

drawing and shaping tools, this turns the cursor into a 
drawing tool, enabling the user to draw freehand 
objects or "brush" the image in any manner. 

20 FIGURE 3 illustrates an exemplary routine for implementing the imaging 

program 21 in accordance with the invention. At block 60, a system startup is 
performed wherein the computer looks for peripheral devices that are connected to 
the aesthetic imaging system, the memory is tested, and any other startup procedures 
needed to get the system up and running are implemented. At block 62, the imaging 

25 program displays a main menu, which provides access to the various features of the 
imaging program. Specifically, the main menu includes the following options: 
Storage, Camera, View, Shape, Draw, Print, Modules, and Exit. Those options that 
are pertinent to the invention are described in further detail below. 

At block 64, a test is made to determine if the Camera option has been 

30 selected from the main menu, indicating that the user wants to take a picture of a 
patient. If the Camera option has been selected, a routine to implement this command 
is called at block 66. A suitable subroutine for this task in illustrated in FIGURE 4. 
Upon return from the Camera routine, the program loops to block 62. 

If the Camera option was not selected, a test is made at block 68 to determine 

35 if the Storage option has been selected for the main menu, indicating that the user 



WO 98/20458 PCT/US97/20394 

-11- 



wants to load an image (picture) from computer storage, e.g., a hard drive. If the 
Storage option was selected, the image(s) to be edited or viewed during the current 
session are selected at block 70. 

At block 72, the selected images are copied to the appropriate buffers in the 
5 frame grabbing board, as described in FIGURE 2 and the accompanying text. For 
example, if the selected image is an original image that has not yet been edited, the 
original image will be copied to the original, modified, and current image buffers. If 
the selected image is an image that has previously been modified, the original image is 
copied to the original image buffer 50 and the modified image is copied to both the 

10 modified and current image buffers 52 and 54. It will be appreciated that the number 
of images that may be loaded at one time will be limited, in part, by the capacity of the 
frame grabbing board. The program then loops to block 62. 

If the Storage option was not selected, a test is made at block 74 to determine 
if the Draw option has been selected from the main menu. If the Draw option has 

15 been selected, a draw tool routine is called at block 76. The program then loops to 
block 62. A suitable routine for implementing the Draw option is shown in 
FIGURE 5. 

If the Draw option was not selected, a test is made at block 78 to determine if 
the Shape option has been selected. If the Shape option has been selected, a shape 
20 tool routine is invoked at block 82. A suitable routine for implementing the Shape 
option is shown in FIGURE 10. Otherwise, at block 82 a test is made to determine if 
the View option has been selected. 

If the View option has been selected, a view subroutine is invoked at block 83. 
An appropriate routine for the View option is shown in FIGURE 13. The program 
25 then loops to block 62. 

If the View option was not selected, a test is made of block 84 to determine if 
Hat Modules option has been selected. If the Modules option has been selected, a list 
of the modules available to the user is provided at block 85. Suitable routines for 
implementing the various modules are shown in FIGURES 22A-22D. If the Modules 
30 option is not invoked, the routine continues to a decision block 86. 

At block 86 a test is made to determine whether the Exit option has been 
selected from the main menu. If not, the program loops to block 62. Otherwise, any 
edits to the image are saved at block 88. At this point in the program, the image in 
the current image buffer 54 is saved to the modified image buffer 52, and the working 
35 buffer 56 is cleared. The program then terminates. 
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I. Taking Pictures of a Patient Using an Inverse Image 

FIGURE 4A illustrates an exemplary user interface that utilizes a video camera 
for acquiring a digital image of a patient. It is noted a scanner or other input device 
may also be used to input an image into the aesthetic imaging system. In 
5 FIGURE 4A, the solid blocks indicate user interface options presented to the user by 
the aesthetic imaging system and the dashed blocks represent system responses to the 
decisions made. At block 100, a patient is positioned in front of the video camera. In 
a preferred embodiment of the aesthetic imaging system, an inverse or "mirror" image 
of the patient's image will be displayed on the monitor, as indicated at block 101. The 
10 inverse image is computed using data from the original image, and is representative of 
how patients see themselves day to day when looking into a mirror. More specifically, 
digital images are comprised of pixels or picture elements. It is known to those skilled 
in the art how digital image pixels may be manipulated to create an image that is the 
inverse of the original. 

15 Displaying an inverse image of a patient is advantageous when taking pre- and 

post-surgical pictures of patients because it allows patients to more easily center or 
otherwise position themselves on the monitor. Without the pixel manipulation, the 
input from a camera or other digital device may create confusion when positioning a 
patient. Under normal viewing, if a patient appears left of center in the monitor, they 

20 are in reality too far to the right. In this instance, a typical patient's reaction is to 
move even further to the right. With a mirror image displayed, the tendency of most 
patients is to naturally adjust to the desired position. 

The displaying of a mirror image is particularly important when taking post- 
surgical pictures. In post-surgical pictures, it is advantageous to have the patient in 

25 exactly the position they were in when taking the pre-surgical picture. For post- 
surgical pictures, the aesthetic imaging system will preferably display a translucent 
inverse image of the pre-surgical picture on the monitor, and then overlay an inverse 
image of the picture currently being taken. As shown in FIGURE 16, a translucent 
patient image 370, in this case a patient's profile, is displayed on the monitor. The 

30 translucent image is the preoperative image taken prior to undergoing a surgical 
procedure. A "live" video image 372 of the patient is also displayed under the 
translucent inverse image. By moving the relative positions of the patient and the 
camera, a user may position the patient in the identical orientation with which they 
took the presurgical picture. When the patient is appropriately positioned, the tip of 

35 the pen may be pressed anywhere on the tablet to freeze the digital image on the 
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monitor. Patients can thus easily align themselves with their former picture to achieve 
very similar before and after pictures. 

The advantage of the translucent method of aligning a patient is shown in the 
side-by-side display shown in FIGURE 17. As shown in FIGURE 17, the aesthetic 
imaging system of the present invention may generate a side-by-side display of the 
two images to allow a patient to easily and accurately compare a presurgical picture 
with a post-surgical picture. The left half of the monitor may display the presurgical 
image 370, and the right half of the monitor may display the post-surgical image 372 
Allowing a patient to view the two images side-by-side in precisely the same 
orientation provides the patient with an accurate impression of the results achieved by 
surgery. 

Upon establishing a desired position for the patient, the image is focused and 
sized at block 102 by using the aesthetic imaging system to adjust the electronic 
controls on the video camera. After any adjustments have been made to the camera, 
at block 104 the tip of the pen is pressed anywhere on the tablet to freeze the digital 
image onto the monitor. At block 106, the user makes a determination if the image 
currently displayed on the monitor is acceptable. If the image is not acceptable the 
routine loops to block 100. If the image is acceptable, an appropriate command is 
entered at block 108 and the image is stored in nonvolatile memory for future 
20 viewing. 

At block 1 10, a test is made to determine if an exit or other similar command 
has been entered by the user, i.e., if any more pictures are to be taken. If additional 
pictures are to be taken, the program loops to block 100. Otherwise, at block 111a 
checksum value (described below) is calculated by the imaging program for each 
25 (original) image that has been stored. At block 1 12, the imaging program stores each 
image and its associated checksum value. The routine then returns to block 68 of 
FIGURE 3. 

E Determining Authenticity TT P i ne Checksum Values 

The checksum value is an addendum to an original stored image that is used to 

30 determine its authenticity when the image is subsequently displayed or printed. Those 
skilled in the art will recognize that there are a number of methods of implementing 
such a checksum procedure. For example, one checksum computation is to add up 
the grayscale values for one of the colors, i.e., red, green, or blue, for each pixel 
comprising the image. Assuming a screen of 640 by 480 pixels and 256 colors per 

35 pixel, the checksum values would range from 0 to (640X480)(255). When an image 
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is recalled for display or to be printed, the checksum value is recalculated. ' If the 
image has not been modified, the newly calculated checksum value will be equivalent 
to the addendum value, and the image is certified as being unaltered. If the image has 
been modified, the checksum values will vary, indicating the image has been modified. 
5 In this instance, an indication of the fact that the image has been modified may be 
displayed or printed with the image, if desired. 

The authentication of an original image using a checksum value is ideal for 
situations in which physicians display before and after pictures of a patient who has 
undergone cosmetic surgery. In some instances, viewers are skeptical as to whether 

10 an "after" image is really representative of a patient's appearance after surgery. This is 
in reaction to beliefs that post-surgical images have been altered to make the patients 
look better. For example, there may be temptation to slightly fade wrinkles or 
otherwise edit features that the physician was attempting to address in a surgery. 
Using the described checksum feature, the post-surgery image can be verified as an 

IS authentic, unaltered image based upon the addendum value, and the veracity of the 
image is not questioned. This is beneficial to physicians when illustrating post- 
surgical results during lectures or in other teaching situations. 

FIGURE 4B illustrates an exemplary routine for determining whether or not 
an original image, i.e., pre- or post-surgical image, has been altered in accordance 

20 with the invention. This routine may be invoked whenever a pre- or post-surgical 
image is displayed on a monitor or printed on a page. At block 1 14, a test is made to 
determine whether the image to be displayed is portrayed as an "original image" that 
was not modified, e.g., a before or after picture. If the image is supposed to be an 
original image, a current checksum value for the image is calculated at block 1 15. At 

25 block 1 16, the calculated checksum value for the image is compared to the checksum 
that was stored when the image was acquired by the aesthetic imaging system, e.g., 
when the picture was taken 

At block 117, the calculated checksum value is compared to the stored 
checksum value to see if they are equivalent. If the two values are not equivalent, at 

30 block 118 an icon is added to the image, e.g., displayed or printed along with the 
image, indicating that the image has been altered. If the checksum values are 
equivalent, an icon is added to the image at block 1 19 verifying the authenticity of the 
image. Once either icon has been added to the image, or if the image being printed or 
displayed is not an original image, the program terminates. As will be appreciated, the 

35 same checksum computational method must be used on each image, i.e., when an 
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original image is acquired and when an image is to be displayed, or the comparison 
will be meaningless. 

While the use of a checksum is contemplated in the exemplary routine for 
determining whether an original image has been altered, it will be appreciated that 
5 other techniques may be used to detect when an original digital image has been 
modified. For example, a flag or other marker may be uniquely associated with the 
original digital image. If the flag or other marker is absent from the digital image 
being displayed, the digital image is a copy that is presumed to have been changed. 
Alternatively, two versions of the image may be stored, included an unaltered original 

10 and a copy. The two images may be compared in order to determine whether 
modifications have been made to the copy. 

The editing aspects of the invention are now described. 
m. Editing Using Draw Tools 

A disadvantage of prior art aesthetic imaging systems is that a physician or 

15 facilitator may have to cycle back and forth between several menus in order to 
properly edit an image. In an effort to minimize the number of menus required, the 
imaging program includes a unique combination draw (CD) feature that generally 
works with all of the drawing tools. The CD feature allows a user to freehand draw, 
use curves to edit an image, as well as undo using either freehand or curves, without 

20 having to invoke a separate menu for each item. 

The following describes an interface for the CD feature as implemented in an 
embodiment of the invention. For purposes of this discussion, it will be assumed that 
the airbrush tool is selected as the drawing tool, although it is to be understood that 
the CD tool generally applies to all of the drawing tools. Upon selection of the 

25 airbrush draw tool, the aesthetic imaging system prompts the user to choose a color 
from a color palette that appears on the monitor. A color is selected using the pen. 
After selecting a color, a side bar menu is displayed. The user may select from a 
number of options on the side bar, including brush size and intensity, or select away 
from the side bar menu to remove the menu from the screen. In the latter case, the 

30 system defaults are used. 

To freehand draw, the user presses on the tablet with the pen tip and continues 
pressure while moving or "rubbing" the pen on the tablet. At this point the chosen 
color is written onto the image at the location on the monitor that corresponds to the 
pen location. Pressing the side bar while repeating the motion will allow the user to 

35 selectively remove any edits to the image using a freehand motion. 
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To draw a curve, the user must set a first anchor point by selecting with the 
pen. Thereafter, as the user moves or "floats" the pen across the tablet, a green line 
will extend from the first anchor point to the current position of the pen. In a desired 
location, a second anchor point is set by selecting with the pen. Once both anchor 
5 points have been established, the green line appears on the monitor as a segment 
between the two anchor points. To create a curve, the user floats the pen across the 
tablet. The system will display a curved line bending and moving with the movement 
of the pen. The pen movement (top to bottom, or side to side) determines the arc of 
the curve. As the curve moves across the image, the image is edited in accordance 

10 with the selected draw tool and draw tool options. With the airbrush example, the 
system displays the curved line repeating itself with the chosen color. Pressing the 
side bar while repeating the motion will allow selective removal of any edits to the 
image using a curve established between the anchor points. 

FIGURE 5 illustrates an exemplary routine for implementing the CD feature of 

15 the imaging program. In one embodiment of the invention, the draw tool group 
includes: Airbrush, Tint, Texture, Blend, Undo, and Contour. Once a draw tool is 
selected from the main menu at block 74 of FIGURE 3, the routine of FIGURE 5 
begins and selections must be made, or default selections confirmed, for the selected 
draw tool. Specifically, at block 120 the user is prompted to enter any options that 

20 are applicable to the draw tool selected. The airbrush tool will be described as an 
example. With airbrush, a color must be chosea After a color has been chosen, a 
side bar menu is displayed at block 122 which illustrates other options that may be 
invoked for the airbrush tool. These typically include selecting a brush size, brush 
intensity, and other miscellaneous options. Sample side bar menus for various shape 

25 tools are described below. The draw tool side bar menus are similar to these. 

Once the side bar menu options are entered, a test is made at block 124 to 
determine if the side button has been depressed. Pressing the side button can mean a 
cancelftack-up or an "undo," depending upon when it is activated. It is noted that at 
any point during the draw routine and the other routines described below, the side 

30 button may be used to return to the main menu through multiple backup or cancel 
commands. 

If the side button has not been pressed, a test is made at block 126 to 
determine if the tip of the pen has been pressed. If the tip has not been pressed, the 
routine loops to block 124. Otherwise, a test is made at block 128 to determine if the 
35 tip was released prior to moving the pen. If the pen has been moved while the pen tip 
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was pressed against the tablet, the imaging program enters freehand draw mode, 
shown at block 130. In one embodiment of the invention, freehand draw mode is 
entered if the pen moves the equivalent of three or more pixels. While in freehand 
draw mode, freehand edits may be made to the image in a manner similar to prior art 
5 imaging programs. This mode will remain until pressure on the pen tip is released. 
After beginning freehand draw mode, the routine loops to block 124. 

If the pen was not moved, e.g., the pen has been moved two pixels or less 
prior to releasing the tip, the imaging program begins a curve mode by establishing a 
first endpoint, as indicated at block 132, and drawing a line on the monitor from the 

10 endpoint to the current pen position, indicated at block 134. At block 136, a test is 
made to determine if the tip has been pressed. The imaging program at this point is 
looking for a second endpoint to be entered. If not, the program loops back to 
block 136 to await the input. As noted above, the user can go back to the beginning 
of the routine using the cancel button if the user has changed his or her mind, although 

15 this is not shown in the flow diagram. Specifically, the first cancel would place the 
routine at block 124, the second at either block 122 or 124, depending on the draw 
tool selected, and subsequent cancels would forward the routine to the main menu. 

Once the tip is pressed, the second endpoint is established at the tip position, 
and a line segment is drawn on the monitor from the two endpoints, as indicated at 

20 block 138. At this point in the routine, the imaging program is in "curve draw mode" 
as indicated at block 140, and the user can make any edits desired using a curvilinear 
line segment having the established endpoints. The routine loops to block 124 while 
in this mode. 

With reference to block 124, if the side button has been depressed a test is 
25 made at block 142 to determine if the imaging program is in either the freehand or 
curve draw modes. If the imaging program is in either mode, an undo mode will be 
invoked as long as the side button remains depressed, as indicated at block 144. If at 
the time of depressing the side button the imaging program is in freehand draw mode, 
the undo will also be freehand. Selective freehand undo edits are thus available. 
30 Similarly, if at the time of depressing the side button the imaging program is in curve 
draw mode, the undo will be in this mode. Selective "undo" edits are then available 
using a curve, as opposed to freehand drawing. Releasing the side button will return 
the imaging program to the drawing mode that was active just prior to depressing the 
side button. 
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Once the side button is released, or if the imaging program was not in either 
drawing mode, a test is made at block 146 to determine if the user wishes to exit. 
This test generally refers to the user again depressing the side bar, and thus "backing 
out" of the drawing routine. In this case, the draw routine returns to block 78 of 
5 FIGURE 3. Otherwise, the routine branches to block 124. 

To further exemplify the advantages of the CD draw tool described in 
FIGURE 5, an original (unedited) image 130 that is representative of an image 
displayed on a monitor is illustrated in FIGURE 6. A main menu 132 is displayed 
across the top of the image to allow a user to select editing, viewing and printing 

10 options, as discussed in FIGURE 3 and accompanying text. The main menu 132 is 
from an embodiment of the aesthetic imaging system 20. 

FIGURES 7 A-7E and 8A-8E contrast exemplary steps taken to make identical 
edits to the image 130. The steps required to modify the image using a relatively 
advanced prior art imaging system are illustrated in FIGURES 7A-7E. These steps 

1 5 are modeled after a prior art imaging system that has been distributed by Mirror Image 
Technology, Inc., a division of Virtual Eyes, Incorporated, the assignee of the present 
invention. The steps required using an embodiment of the aesthetic imaging 
system 20 in accordance with the invention are illustrated in FIGURES 8A-8E. 
Briefly, each set of drawings illustrates examples of edits to a patient's nose, chin, and 

20 neck regions. The edits are for use in explaining the invention only, and merely 
exemplary in nature. 

A brief description of the edits follows. With reference to FIGURE 7 A, the 
curve option of an airbrush tool is used to modify the bridge of a patient's nose. A 
resultant curve 134 is displayed having anchor points (endpoints) 136 and 138. In 

25 FIGURE 7B, a freehand motion is used to eliminate a portion of the tip of the 
patient's nose. In FIGURE 7C, an undo tool is used to replace a portion of the bridge 
of the patient's nose that was removed by the edits performed in FIGURE 7 A In 
FIGURE 7D, the neck area of the patient has been edited using the curve option of an 
airbrush tool. The resultant curve 140 has anchor points 142 and 144. Finally, in 

30 FIGURE 7E, an undo tool is used to add back a portion of the neck area that was 
removed in FIGURE 7D. These edits require the following steps: 



FIGURE 7A: Step Si 
Step S2 
Step S3 



move pen to draw on main menu; 

select draw; 

move pen to airbrush; 
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StepS4 
Step SS 
Step S6 
Step S7 

5 Step S8 

Step S9 
Step S10 

10 

Step Sll 
Step S12 

15 FIGURE 7B: Step S13 

Step S14 
Step S15 
Step S16 

20 Step S17 

Step SI 8 
Step S19 
Step S20 
Step S21 

25 

FIGURE 7C: StepS22 

Step S23 
Step S24 

30 Step S25 

Step S26 
Step S27 
Step S28 
Step S29 



select airbrush; 

move pen to curve; 

select curve; 

select an airbrush color; 

move pen to the first anchor point 

position; 

select at the position to establish the 
anchor point 136; 

move pen to the second anchor point 
position; 

select at the position to establish the 
anchor point 138; 

move pen to bend the curve 134 into the 
bridge of the nose; 

press the side button on the pen to exit 

to the main menu; 

move pen to draw; 

select draw; 

move pen to airbrush; 

select airbrush; 

move pen to freehand; 

select freehand; 

select a color for the airbrush tool; 

use a rubbing motion with the pen to 

make the freehand edit; 

press the side button on the pen to exit 

to the main menu; 

move pen to draw; 

select draw; 

move pen to undo; 

select undo; 

move pen to curve; 

select curve; 

use a rubbing motion with the pen to 
undo the previous edit; 



WO 98/20458 



-20- 



PCT/US97/20394 * 



FIGURE 7D: StepS30 

Step S31 
Step S32 

5 Step S33 

Step S34 
Step S35 
Step S36 
Step S37 

10 

Step S38 
Step S39 

15 StepS40 

Step S41 
Step S42 

20 

FIGURE 7E: StepS43 
Step S44 

25 Step S45 

Step S46 
Step S47 
Step S48 
Step S49 

30 Step S50 

Step S51 



press the side button on the pen to exit 

to the main menu; 

move pen to draw; 

select draw; 

move pen to airbrush; 

select airbrush; 

move pen to curve; 

select curve; 

select color to be used by the airbrush 
tool; 

move pen to the first anchor point 
position; 

select at the position to establish the 
anchor point 142; 

move pen to the second anchor point 
position; 

select at the position to establish the 
anchor point 144; 

move pen to bend the curve 140 toward 
the neck, thereby making the edit 
shown. 

press the side button on the pen to exit 

to the main menu; 

move pen to draw; 

select draw, 

move pen to undo; 

select undo; 

move pen to freehand; 

select freehand; 

use a rubbing motion with the pen to 
undo a portion of the previous edit; and 
press the side button to exit back to the 
main menu. 
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In accordance with the invention, the steps required to perform the same edits 
using the aesthetic imaging system 20 are now described. With reference to 
FIGURE 8 A-8E, the steps required to perform the edits include: 



10 



15 



20 



FIGURE 8A: StepNl 
Step N2 
Step N3 
Step N4 
Step N5 
Step N6 

Step N7 

Step N8 

Step N9 

Step N10 

FIGURE 8B: Step Nil 

FIGURE 8C: StepN12 



25 



30 



FIGURE 8D: Step Nil 
Step N12 
Step N13 
Step N14 



move pen to draw; 
select draw; 
move pen to airbrush; 
select airbrush; 

elect any color for the airbrush tool 
move pen to the first anchor point 
position; 

select at the position to establish the 
anchor point 136; 

move pen to the second anchor point 
position; 

select at the position to establish the 
anchor point 138; 

move pen to bend the curve 134 into the 
bridge of the nose; 

pressing the tip of the pen against the 
tablet and use a rubbing motion to make 
the freehand edit. 

pressing the tip of the pen and the side 
button simultaneously, and maintain 
pressure while rubbing in the area to be 
undone; 

move pen to the first anchor point 
position; 

select at the position to establish the 
anchor point 142; 

move pen to the second anchor point 
position; 

select at the position to establish the 
anchor point 144; 
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Step N1S move pen to bend the curve 140 toward 
the neck, thereby making the edit shown. 
FIGURE 8E: Step N16 press tip of pen and the side button 

simultaneously, while rubbing to undo a 
5 portion of the previous edit; and 

StepN17 release pressure on the pen and side 
button, and press the side button to 
return to the main menu. 
From the simplified edits shown in FIGURES 7A-7E and FIGURES 8 A-8E, it 
10 will be apparent that the aesthetic imaging system 20 provides a distinct advantage 
over prior art systems. Specifically, patients find the continued back and forth 
motions required to select necessary tools from the main menu to be disconcerting. 
The aesthetic imaging system 20 simplifies the editing process by providing freehand, 
curve, and undo options in the pen commands themselves, instead of in separate pull- 
15 down menus as is done in the prior art. The combination of tools is extremely 
effective in performing edits quickly, efficiently, and nearly seamlessly, all of which 
benefit both patient and physician during the consultation process. 

FIGURE 9A illustrates a user interface for a contour tool for use in editing 
images in accordance with the invention. The contour tool is invoked from the main 
20 menu of the imaging program, as indicated at block 200. The contour tool has 
similarities to a blend tool, but utilizes pixel manipulation to pull pixels from one area 
to another. For example, the tool works great for chin and lip augmentations. 

After the contour tool has been selected, a side bar menu is displayed by the 
aesthetic imaging system, as indicated at block 201. At block 202, the point size for 
25 the tool may be selected from the side bar menu. At block 203, an opacity percentage 
is entered by the user. If the opacity is at 100 percent, any areas affected by the edit 
are completely covered by the replacement pixels. As the percentage is reduced, more 
and more of the original pixels will remain, creating a blending of the replacement and 
prior pixels. 

30 Anchor points are selected at block 202. The selection is accomplished as 

described in blocks 128 and 130 of FIGURE 5. As also described, once the anchor 
points are selected, a line is displayed between the points by the aesthetic imaging 
system, as indicated at block 205. At block 204, a curve having the anchor points as 
endpoints is positioned along a part of the body, e.g., lips or chin, to be edited. The 

35 body part is then edited by dragging the curve in the direction in which a body part is 
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to be augmented. Edits made in block 206 are saved at block 208 by pressing and 
then releasing the tip of the pen. The program then terminates. 

FIGURES 9B-9C further describe the operation of the contour tool, by 
illustrating how pixels are replicated from one area of an image to another. The image 
5 areas described are for exemplary purposes only, and are simplified for clarity in this 
discussion. With reference to FIGURE 9B, an area 209 of an image is comprised of 
red218R, green218G, blue218B, and yellow 21 8 Y areas separated by boundary 
lines 210, 212, 214, and 216. It is assumed that a pair of anchor points 218 and 219 
have been established by a user along the boundary 212, wherein the aesthetic imaging 
system will display a line segment 220 between the two anchor points. 

In FIGURE 9C, it is assumed that the user has moved the midsection of the 
line segment 220 to the right. In this instance, the blue area 218B has been stretched 
into the yellow area 2 18 Y. This area is bounded by the line segment 220 (now 
curved) and the boundary line 214. Also, the green area 218G has been stretched into 
the blue area218B. This area is bounded by a curved line segment 221 and the 
boundary line 212. The red area 218R has expanded into the green area 218G; this 
area is bounded by a curved line segment 222 and the boundary line 210. 

If the opacity level is at 100 percent, the newly defined areas will be comprised 
of the color being expanded. Thus, the area bounded by segments 220 and 221 will 
be blue; the area bounded by segments 221 and 222 will be green; and the area 
bounded by segment 222 to the left edge of the diagram will be red. 

If the opacity level is less than 100 percent, pixels from the underlying image 
areas that are being written over by the newly defined areas will be blended into the 
newly defined areas. At an opacity of 80 percent, for example, the area bounded by 
segments 220 and 221 will still be primarily blue, but the portion of this area bounded 
by the segment 220 and the boundary line 214 may have a yellow tinge; and the 
portion of this area bounded by the boundary line 212 and segment 221 may have a 
green tinge. As the opacity percentage is dropped, the effects on these areas will be 
even greater. 

While somewhat simplistic, the illustration in FIGURES 9B and 9C describes 
the function of the contour tool and the effect on more complicated pixels patterns 
will be appreciated by those skilled in the art. The area bounded by segment 220, 
boundary line 214, segment 221, and boundary line 212 will not be affected by 
changes in the opacity setting. 
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FIGURES 9D-9G illustrate edits to a patienfs lips 224 using the contour tool. 
FIGURE 9D is a "before" picture without any modifications. To edit the right half of 
the patient's upper lip, the user first selects the desired end points surrounding the 
feature to be modified. A first anchor point 374 is designated near the middle of the 
5 upper lip, and a second anchor point 376 is designated at the right outer margin of the 
upper lip. When the anchor points have been selected, a line 378 is displayed between 
the points by the aesthetic imaging system. By floating the pen over the tablet in a 
direction generally indicated by arrow 380, the line is bent to form a curve 382 that 
approximates the contour of the feature being edited. When the user has fitted the 

10 curve to the feature, the shape of the curve is set by tipping the pen. As shown in 
FIGURE 9E, the user then selects a point along the portion of the curve by tipping the 
pen at the desired location. By floating the pen above the tablet, the user can stretch 
the selected feature in the manner described above. In FIGURE 9E, the user floats 
the pen in a direction generally indicated by arrow 384 to enlarge the patient's right 

15 upper lip. As the upper lip is expanded, the pixels forming the upper lip are replicated 
to expand the portion of the lip, while the pixels outside of the upper lip are deleted. 

In FIGURE 9E, the left side of the person's upper lip has been edited using the 
contour tool In FIGURE 9F, both the left and right sides of the person's upper lip are 
shown modified using the contour tool. In FIGURE 9G, the lower lip has been 

20 edited. Using the contour tool, these edits are accomplished very quickly, in part 
because the augmented area will automatically match the area around it. The features 
of the contour tool make it much easier to perform augmentations than is currently 
available using prior art imaging tools. 
IV. Editing Using Shape Tools 

25 FIGURES 10-12B are directed to the shape tool features of the aesthetic 

imaging system. Similar to the draw tool, the imaging program has a combination 
feature that is generally available with any of the shape tools. This feature, called the 
autoblend feature, allows the user to easily move and paste shape tool edits, with or 
without blending the edges around the edit. 

30 FIGURE 10 illustrates an exemplary routine for implementing the autoblend 

feature of the imaging program. The shape tool group generally includes: Warp, 
Stretch, Copy, Cutout, Rotate, Freeze Compare, and Resize. The routine of 
FIGURE 10 is invoked after a shape tool has been selected at block 80 of FIGURE 3. 
The side bar menu for the selected shape tool is displayed at block 230. Exemplary 

35 side bar menus for various shape tools are set forth below. The side bar menus are 
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e!^^ ,? *u° deta " ° Pti ° nS for 3 *» tool, and may be 

especially helpful for those unfamiliar with imaging packages. 



Copy, Cutout, Resize Stretch, Rotate Freeze Compare 



Zoom 
Undo 



Zoom Zoom 



Undo Undo 
Compare Compare Splitimage 

Split image Splitimage 
Inverse Inverse 
B,end Blend 
The black dots are brush size options that allow a user to choose the thickness 
of a shaping tool. The zoom option allows a user to • • uieuucKness 

™- v ^ luws a user to look at an image in greater detail 

When he zoom op„on b invoked, the aesuteuc imastag ^ „ £ . 
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At block 232, the user is prompted to designate an area of the image to be 
edited. In a preferred embodiment, this is accomplished by pressing down on the pen 
and freehand drawing an area, e.g., a circle, that is to be subject to the edit. In this 
regard, the imaging program contains a unique feature wherein if a partial area is 
5 designated and the pen subsequently released, the drawing area will automatically be 
formed into a contiguous area by the imaging program. At block 236, a test is made 
to determine if an area has been designated by the user. If not, the routine loops back 
to block 234 and awaits a designation. 

After an area has been designated, any edits to the designated area of the 

10 image are performed in accordance with the selected shape tool, as indicated at 
block 236. Two exemplary shape tools for editing an image are illustrated in 
FIGURES 12A and 12B. At block 238, a test is made to determine if editing of the 
designated area is complete. In one embodiment, this involves testing for when the 
user "selects" with the pen anywhere on the tablet. The routine remains at block 238 

1 5 until editing is complete (or the user exits using the side button). Upon completion of 
the edits, an autoblend box is displayed in the vicinity of the edited area, as indicated 
at block 240. 

At block 242 a test is made to determine if the tip of the pen has been pressed 
against the tablet. If not, the routine loops, testing for this occurrence (or an exit 

20 command from the user). After the pen tip has been pressed, the imaging program 
calculates the location of the pen relative to the autoblend box. 

FIGURE 1 1 illustrates an example of an autoblend box 250 that may be drawn 
by the aesthetic imaging system in accordance with the invention. As is discussed 
above, the autoblend box 250 may be used to: (1) move an edited area, (2) paste the 

25 edited area onto the image while blending the edge created between the edited area 
and the rest of the image, and (3) paste the edited image without blending. While the 
autoblend box 250 uses the conventions set forth below, those skilled in the art will 
appreciate that other conventions may be used without departing from the scope of 
the invention. 

30 To move an edited area, the user must press down on the pen anywhere inside 

the autoblend box, except not at the three-, six-, nine-, and twelve-o'clock positions of 
the box. The "move" area is designated with reference numeral 254. To paste the 
edited area without blending, the user must press within the outlined areas 254 at the 
three-, six-, nine-, and twelve-o'clock positions inside the box. Selecting anywhere 

35 outside the box, i.e., area 256, results in the edited area being pasted with blending. 
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With reference again to FIGURE 10, a test is made at block 244 to determine 
if the pen tip has been pressed at a location outside of the autoblend box. If the pen 
tip was pressed at a location outside the autoblend box, the edit area is pasted with a 
blending of the edges at block 260. Otherwise, a test is made at block 262 to 
5 determine if the tip location was within the approximate three-, six-, nine-, and 
twelve-o'clock areas of the autoblend box. A pressing of the tip within any of these 
areas results in the edited area being pasted without blending, as indicated at 
block 264. As indicated at block 266, a selection in a location in the autoblend box 
apart from the three-, six-, nine-, and twelve-o'clock areas will allow the image to be 

10 moved. In this case, the edited area will track movement of the pen as long as the tip 
remains pressed. After a move is completed, the routine loops to block 242. 

If a paste has been accomplished using blocks 260 or 264, a test is made at 
block 268 to determine if the user wishes to exit the shape routine, e.g., by pressing 
the side button. If not, the routine loops to block 232 where a new area of the image 

15 may be considered. Otherwise, the routine returns to block 82 of FIGURE 3. 

FIGURES 12A and 12B illustrate two exemplary shape tools that are available 
when using the aesthetic imaging system. With reference to FIGURE 12A, a cutout 
tool is unique in that a user can select an area of the image to be cut out, thereby 
creating a "hole" in the image, and an identical image underneath the cutout image can 

20 then be moved in all directions as it is viewed through the hole. The cutout feature is 
especially useful for profile views including chin augmentation, brow lifts, and 
maxillary and mandibular movement; and frontal views, including otoplasty, brow lift, 
lip augmentation, nasal base narrowing, and maxillary and mandibular movement. 

At block 270, the current image is copied to a working buffer. As is discussed 

25 in FIGURE 10, when the cutout subroutine is called the user has defined an area of 
the image to be edited. At block 272, a boundary is created around the designated 
area designated in block 234 of FIGURE 10. At block 274, the working area is 
displayed inside the boundary, and the current image displayed outside the boundary. 
In this manner, the image in the working buffer can be moved relative to the image in 

30 the current buffer until the desired alignment has been achieved. The program then 
returns to the routine of FIGURE 10. 

Upon returning to block 238, the edit may be frozen by selecting with the pen. 
Thereafter, the autoblend box is displayed. Selecting within the area 252 of the 
autoblend box allows the designated area to be moved. Selecting anywhere outside 

35 the autoblend box will make the edit permanent, with automatic blending. Selecting 
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within the box at the three-, six-, nine- or twelve-o'clock positions (areas 254) will 
make the edit permanent, without blending. 

With reference to FIGURE 12B, a rotate tool in accordance with the invention 
is particularly useful when editing profile views including the nasal tip, mandible, 
5 maxilla and brow areas; and frontal views including nares, brows, and the comers of 
the mouth. As is discussed in FIGURE 10, when the rotate routine is called, the user 
has defined an area of the image to be edited. At block 276, the area designated in 
block 234 of FIGURE 10 is shown in phantom. At block 278, the imaging program 
waits for the user to enter an axis of rotation. An axis is then entered by the user, as 
indicated at block 279. 

Once an axis of rotation is entered, a display line emanating from the axis point 
is displayed on the monitor, as indicated at block 280. Also, the number of degrees of 
rotation is displayed. The position of the pen dictates the degree of rotation. As the 
pen is moved away from the axis point, the display line will lengthen, providing the 
user greater control of the rotation of the designated area. At block 282, the system 
waits for the user to enter a desired degree of rotation. The degree of rotation is 
entered by the user by selecting with the pen, as indicated at block 283. Once the 
degree of rotation is entered, the designated area is redrawn onto the current image, 
as indicated at block 284. 

After the redraw, the routine returns to FIGURE 10. Upon returning, the 
autoblend box is displayed. Selecting within the autoblend box allows the designated 
area to be moved. Selecting anywhere outside the autoblend box will make the edit 
permanent, with automatic blending. Selecting within the box at the three-, six-, 
nine-, and twelve-o'clock positions will make the edit permanent without blending. 

While prior art imaging programs have a rotate feature, they do not allow a 
user to select the axis of rotation. The ability to select the axis is valuable in the 
procedures listed above. 

FIGURES 18A-18D disclose the capabilities of the warp shape tool in the 
aesthetic imaging system. The warp shape tool is a powerful tool that allows users to 
easily edit a patient's features. Similar to the contour tool, the warp tool allows a user 
to select and manipulate a feature of the patient's image, with the aesthetic imaging 
system automatically redrawing the area surrounding the manipulated feature as the 
manipulation is being performed. 

As shown in FIGURE 18 A, the user first defines an image to be manipulated 
by the warp tool by encircling the portion of the image to be edited as shown by 
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dotted line 386. Once the area has been selected, a user may tip the pen to designate 
a stretch point within the selected area. As shown in FIGURE 18 A, a stretch 
point 388 has been designated near the top portion of the patient's right upper lip. 
Once the stretch point has been designated, the user may float the pen in the desired 
5 direction that they would like to stretch the image. The portion of the image that is 
located at the stretch point is pulled in the direction that the user floats the pen, with 
the area surrounding the stretch point either expanded or compressed. That is, the 
area in the direction that the user floats the pen is compressed, and the area away from 
the direction that the user floats the pen is expanded. The amount of expansion or 

10 compression is dictated by the distance of the area away from the stretch point, in a 
manner discussed in greater detail below. Preferably, the image is manipulated in real 
time, so that the user is presented with a seamless and continuous stretching or 
movement of the selected feature. 

The technique used to implement the warp tool is portrayed in 

15 FIGURES 19A-19E. A user first defines a manipulation area 420 to be edited by 
circling the desired feature of the patient. As shown in enlarged detail in 
FIGURE 19B, when the manipulation area has been selected, the aesthetic imaging 
system creates a pixel map of a warping area 422 that completely encompasses the 
manipulation area. For computational purposes, the warping area is preferably 

20 rectangular and approximately 33% larger in area that the manipulation area. It will 
be appreciated, however, that the warping area may be differently shaped or sized 
depending on the particular application and available system hardware. 

After defining the manipulation area, the user selects a stretch point 424 within 
the manipulation area by tipping the pen at a desired location within the area. As 

25 shown in FIGURE 19C, when the stretch point is selected, the aesthetic imaging 
system maps four rectangles 426a, 426b, 426c, and 426d in the warping area. One 
corner of each rectangle is defined by the stretch point, and the diagonally opposing 
corner is defined by a corner of the warping area. Perpendicular lines 427 drawn 
through the stretch point intersect the warping area boundary at points 428a, 428b, 

30 428c, and 428d. After the stretch point is selected, the user may float their pen to 
manipulate the selected feature, for example in the general direction indicated by 
arrow 430. 

As the user floats the pen to warp the selected feature, the rectangles in the 
warping area are distorted. In FIGURE 19D, the stretch point has been moved 
35 upwards and to the left in the warping area, to an intermediate location 432. 
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Periodically during the stretch point's motion, the aesthetic imaging system 
determines the shape of four quadrilaterals 434a, 434b, 434c, and 434d, with sides 
that extend from the intermediate location 432 of the stretch point to original 
points 428a, 428b, 428c, and 428d. Using a bilateral transformation, the pixels in the 
5 original rectangles 426a, 426b, 426c, and 426d are then mapped into the 
quadrilaterals. Further movement of the stretch point repeats the process, the original 
set of rectangles being mapped into the new quadrilaterals that are generated by the 
movement. If the display is updated frequently as the stretch point is moved, the user 
views the mapping and remapping as a warping of the patient's feature within the 

10 manipulation area. It will be appreciated that only the portion of the warping area that 
coincides with the manipulation area is presented to the user. The pixels comprising 
the remaining portion of the warping area are used for computational purposes, but 
are not displayed to the user. 

When the desired warping effect is achieved, the user tips the pen to fix the 

15 stretch point in a desired location. To ensure there is no discontinuity between the 
manipulation area and the surrounding area of the patient's image, automatic blending 
around the outer margins of the manipulation area is performed by the aesthetic 
imaging system. As shown in FIGURE 19E, the aesthetic imaging system also remaps 
the warping area, creating four new rectangles 436a, 436b, 436c, and 436d based on 

20 the ending location 438 of the stretch point. In a preferred embodiment of the 
invention, a second stretch point may then be selected within the manipulation area 
and the process repeated, with the second warping transforming rectangles 436a, 
436b, 436c, and 436d that resulted from the first warp. After the second warping is 
completed, the user may then select a third or additional warp point to further 

25 manipulate the image. Each manipulation is performed without the user having to 
redefine the manipulation area. 

The flexibility and power of allowing multiple warps on a selected region is 
demonstrated in FIGURES 18A-18C. Returning to FIGURE 18A, a user designates a 
stretch point 388 and floats the pen in a direction generally indicated by arrow 390. 

30 Floating the pen upwards and outwards generally causes the upper right portion of the 
lip to be expanded in that direction. The resulting manipulation is shown in 
FIGURE 18B, wherein the right lip has been made fuller. The area surrounding the 
lip is expanded or compressed to ensure that there are no discontinuities between the 
edited lip and the surrounding face. In FIGURE 18B, the user selects a second stretch 

35 point within the manipulation area to further stretch the selected feature. For 
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example, a stretch point 392 may be positioned on the upper portion of the left lip, 
and the pen floated in a direction generally designated by the arrow 394. As shown in 
FIGURE 18C, this generally causes the upper left portion of the lip to be made fuller 
to match the upper right portion of the lip. Again, during the warping the aesthetic 
5 imaging system automatically expands or contracts the surrounding unmanipulated 
area to ensure that there are no discontinuities between the upper left and the unedited 
portion of the face. 

The warp tool with multiple stretch points is a very powerful tool as it allows 
the user to quickly manipulate an image with a minimum use of drawing tools or 

10 piecemeal editing. Because the warp tool performs the manipulation in real-time, the 
edits are accomplished very quickly and fluidly. A user may therefore generate a 
desired image in a minimal amount of time. 

While the warp tool described above requires the user to define the 
manipulation area 420 to be edited prior to the creation of a warping area by the 

15 imaging system, it will be appreciated that the warping area can alternatively be 
automatically defined by the aesthetic imaging system In such an alternative system, 
rather than defining a manipulation area by circling a region of the image, the user 
simply selects a stretch point 424 and floats the pen in the desired warping direction. 
The imaging system automatically defines a rectangular warping area 422 of a 

20 predetermined size that surrounds the stretch point. As the user moves the stretch 
point, the area within the warping area is warped. Moreover, the size of the warping 
area 422 may also be dynamically adjusted by the imaging system. The farther the 
user floats the pen in a desired warping direction, the greater the imaging system sizes 
the warping area 422. A greater area of the patient's image will therefore be subject 

25 to the warping as the stretch point is pulled a farther distance. Automatically defining 
and dynamically adjusting the warping area allows the user to quickly and easily warp 
an image. 

V. Viewing an Image 

FIGURE 13 illustrates an exemplary routine for implementing the view 

30 features of the imaging program. In FIGURE 13, the solid blocks indicate user 
interface options presented to the user by the aesthetic imaging system and the dashed 
blocks represent system responses to the decisions made. The view group includes: 
Compare, Prioritize, Split Image, Mirror Image, and Restore to Original, as well as 
other options including Zoom and Emboss. At block 300, a test is made to determine 

35 if the Compare option has been selected. 
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The Compare option allows a modified image to be compared to the original 
image so that a viewer can more readily see the changes. Specifically, as the pen is 
floated from the top to the bottom of the tablet, the user will see one image transition 
or "morph" into the other. The morphing is accomplished by overlaying the original 
5 image with the modified image, and varying the opacity of the modified image. When 
the modified image is opaque, only the modified image may be viewed by a user. 
When the modified image is completely transparent, only the original image may be 
viewed by a user. In between these two extremes, varying amounts of feed back the 
edits made to the image will be apparent to the user. The feedback to the patient as 

1 0 the original image morphs into the modified image is much more powerful than a side- 
by-side comparison of the two images. 

When a desired comparison level is achieved, a user can press the tip of the 
pen to freeze an image displayed on the monitor at a point anywhere from zero to 
100% of the transition from the original to the modified image. Freezing an image at 

1 5 a partial transition is extremely helpful where edits have been performed on an image 
that are not realistically achievable in surgery, but an achievable result lies somewhere 
between the original and the modified image. For example, it is easy to edit a blemish 
on a face so that area resembles the surrounding skin and thus becomes invisible. 
However, the total removal of the blemish may not be realistic. In this case, a 

20 transition of that area toward the original image will slowly "fade in" the blemish A 
physician may then freeze the fading process at a desired point to provide a realistic 
image of what surgery can achieve to the patient. 

Another option that is a subset of the Compare option is the Prioritize option. 
The Prioritize option allows a user to designate areas that have been edited so that the 

25 user can selectively illustrate the effects of two or more procedures that have been 
shown to a patient. For example, FIGURE 14A illustrates a modified profile 
image 302 of a patient that includes a rhinoplasty procedure (nose) 304, a chin 
augmentation procedure 306 and a submental lipectomy procedure (neck) 308. The 
boundaries that have been edited are illustrated by dashed lines 304a, 306a, and 308a, 

30 corresponding to the patients original nose profile, chin, and neck, respectively. Using 
the Prioritize option, the user can designate one area on the modified image, and 
illustrate transitions between the original and modified images at that area only by 
floating the pen. Any areas not selected will continue to be displayed as the original 
image. 
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With reference again to FIGURE 13, if the Compare option has been selected, 
a test is made at block 309 to determine if the entire image is to be compared or only 
certain portions of the image, i.e., using the Prioritize option. If less than the entire 
image is to be compared, the user is prompted to enter the area or areas that are to be 
5 compared at block 310. A user may then define one or more "priority areas" by 
freehand circling the desired area. When the priority areas or defined, or if all of the 
edits are to be reviewed during the comparison, at block 31 1 the user is prompted to 
float the pen in a vertical motion on top of the tablet to transition between the original 
and modified images, in accordance with the Compare feature discussed above. An 

1 0 illustration will clarify this point. 

In FIGURE 14B, a first priority area 312 has been defined that corresponds 
generally to the nose. Given this selection, the nose area only will transition from 
original to modified as the pen is moved, with the rest of the image being displayed 
unedited. Thus, the modifications to the chin and neck no longer are shown. In 

15 FIGURE 14C, a second priority area 314 has been defined that corresponds generally 
to the chin. The first priority area 3 12 has been kept. Given these selections, the nose 
and chin areas only will transition from original to modified as the pen is moved, with 
the rest of the image being displayed unedited. Thus, the modifications to the neck 
are not illustrated. 

20 With reference to FIGURE 14D, a third priority area 316 has been defined 

that corresponds generally to the neck, along with the former designations. Given 
these selections, all three priority areas 3 12, 3 14, and 3 16 transition with movement of 
the pen. Again, the undesignated portions of the image are displayed in an unedited 
form, even if parts of the image outside the priority areas have been edited (no edits 

25 are shown). Because edits have not been made, from a user's standpoint the transition 
in FIGURE 14D appears to be a comparison between the original and modified 
images. 

While preferably the user defines the priority areas on the image being edited, 
it will be appreciated that the priority areas may also be automatically defined by the 

30 aesthetic imaging system. A comparison may be made between an original image 
stored in a buffer and the edited image that has been modified by the user. Any areas 
containing differences between the original and the edited image may be highlighted 
by the aesthetic imaging system, and a priority area automatically defined for each 
area containing differences. Whether a priority area is defined may also depend on the 

35 number of differences between the original image and the edited image. 
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In an alternative embodiment of the invention, when a user defines one of the 
priority areas in the Prioritize option, the user is prompted to enter a textual identifier 
for the defined area. For example, after circling the area in FIGURE 14B and defining 
a first priority area 312, the user would be prompted to enter a textual identifier 
5 corresponding to the first priority area. The user may assign a descriptive identifier 
related to that priority area, such as "nose", or the user may assign a non-related 
textual identifier such as "areal". Textual identifiers assigned by the user in this 
manner are displayed to the user in a submenu of the Prioritize option. The user may 
then select from the submenu of the Prioritize option those areas that they wish to 

10 display from the list presented to them. For example, the prioritize submenu 
corresponding to FIGURE 14D may present the user with a choice of "nose," "chin," 
and "neck." By selecting the nose and neck from the prioritize submenu, the user 
could simultaneously display only the effects of the nose and neck procedures. After 
making such a comparison, the user could deselect neck from the prioritize submenu, 

15 and instead select the chin and nose. This would allow the user to display only the 
effects of the chin and nose procedures using the compare option. It will be 
appreciated that assigning textual identifiers to each of the defined priority areas 
provides greater flexibility to a user, since the user does not have to redefine each of 
the priority areas that are to be displayed each time. A user may therefore quickly 

20 cycle through various permutations of the procedures that have been edited for patient 
display. 

In the alternative embodiment in which textual identifiers are assigned to each 
of the priority areas, it will be appreciated that various methods can be used to display 
to the user the correspondence between the textual identifier and the priority area. 

25 For example, when the user selects one of the textual identifiers, the priority area 
could be highlighted or otherwise encircled with a contrasting color to indicate to the 
user the area of the image that corresponds to that textual identifier. Similarly, a user 
may also point to specific areas of the displayed image and have the textual identifier 
corresponding to that area appear to the user. Displaying the correspondence 

30 between priority area and identifier would allow a user to rapidly determine the 
available priority areas that may be shown to a patient when the user has not recently 
worked with or otherwise viewed the image. 

With reference again to FIGURE 13, at block 320 a test is made to determine 
if the user wishes to save a transitional or morphed view of an image. If a transitional 

35 view is to be saved, the user may establish the percentage transition, i.e., anywhere 
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from zero to 100 percent transition (zero percent being the original image and 100 
percent being the edited image), by floating the pen up or down above the tablet to 
establish the view, and the pressing the tip of the pen against the tablet to freeze the 
transitional image, as indicated in block 322. If the tip is pressed again, the frozen 
5 image is saved. The save options are available with or without the priority areas in 
effect. After the save has been accomplished, or if the user did not wish to save a 
transitional view of an image, the Compare option is complete and the routine 
branches to block 326. 

In an alternative embodiment of the invention, the Compare option allows a 

10 user to compare a modified image with any edits made to the modified image during 
the current editing session, i.e., before the changes are permanently saved to the 
modified image. Specifically, with reference to FIGURE 2, this embodiment of the 
Compare option contrasts the image in the current image buffer 54 with the image in 
the modified image buffer 52. As discussed above, this embodiment of the Compare 

15 option may also be used in conjunction with the Prioritize option to allow the user to 
select priority areas for comparison. In this case, the priority areas transition from the 
modified to the current image, while the modified image only is displayed in the other 
(nonselected areas) areas. 

At block 326, a test is made to determine whether the user wishes to view a 

20 split image. The Split Image option is used on a frontal picture only, and allows a 
patient to see his or her asymmetries. If a split image view is desired, the user is 
prompted to select an image, e.g., original or modified, at block 330. At block 332, a 
vertical centerline is displayed on top of the selected image. The user is then 
prompted to position the centerline at the location desired, as indicated at block 334. 

25 Typically, the centerline will be positioned to dissect the image into equal halves, 
using the nose and the eyes as reference points. At block 336, the aesthetic imaging 
system displays two images, one showing the left halves pieced together and the other 
the right halves pieced together. Specifically, the aesthetic imaging system will 
produce an inverse image of the left (right) half and then add it to the left (right) half. 

30 FIGURES 15A-15C illustrate the resultant images that are displayed when the 

Split Image option is invoked. In FIGURE 15A, a frontal image 350 of a patient is 
shown, including a centerline 352 that has been positioned at the center of the patient 
by a user. FIGURE 15B is an illustration of the left halves of the image after being 
pieced together by the aesthetic imaging system, as indicated by reference 

35 numeral 354. FIGURE 15C is an illustration of the right halves of the image, as 
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indicated by reference numeral 356. With the Split Image option, patients can view 
what they would look like if their faces were symmetrical. The tool is especially 
useful in the consultation stage because many people do not realize that the typical 
face is asymmetrical, and changing a face to be perfectly symmetrical, if possible, is 
5 not necessarily desirable. 

Once the split images have been displayed, or if the Split Image option was not 
selected, a test is made at block 360 to determine if the user wishes to view an inverse 
or mirror image of a picture, e.g., to show patients the view they see of themselves 
when looking into a mirror. If an inverse image is desired, the user is prompted to 

10 select an image to be viewed at block 362. The selected image is then "flipped" using 
the aesthetic imaging system, and displayed on the monitor, as shown at block 364. 
After the image has been displayed, or if the inverse image option was not invoked, 
routine returns to block 86 of FIGURE 3. 

When viewing an image on the monitor, it is important that a patient remains 

15 focused on the image being manipulated rather than on the features of the aesthetic 
imaging system. When certain viewing options are selected, for example, the 
prioritize option shown in FIGURES 14A-14D, or the split image option shown in 
FIGURES 15A-15C, the menu bar normally located across the top of the display is 
therefore removed. Only the image of the patient is kept on the screen, ensuring that 

20 the patient remains focused on the image being manipulated. 

In certain situations where it is necessary to display a menu bar, however, it is 
advantageous to minimize the visual appearance of the menu bar. FIGURE 20 is a 
representative image 410 of a patient with a menu bar 412 located across the top of 
the screen 414. To reduce the distraction caused by the menu bar, the menu bar is 

25 preferably translucent to allow the user to view the patient's image through the menu 
bar. The text 416 and the line 418 indicative of the menu bar are preferably presented 
in a contrasting, yet muted, color to allow the user to read the commands. For 
example, the text and the line outlining the menu bar may be presented in an off-white. 
While editing the image with a patient present, the patient is therefore not overly 

30 distracted when the menu bar periodically appears at the top of the screen. Even 
when the menu bar is present for extended periods of time, a patient is not distracted 
since it does not visually stand-out from the patient's image. At the same time, a 
skilled user is provided with sufficient information about the menu choices to allow 
them to choose the appropriate menu options. 
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To further reduce the distraction caused by the menu bar, preferably the menu 
bar 412 may be moved by a user to different locations on the screen 414. For 
example, the menu bar may be moved by a user to the bottom of the screen. 
Particularly when editing the face of a patient on the screen, the majority of the editing 
5 will take place on the upper two-thirds of the screen. Locating the menu bar on the 
bottom of the screen therefore positions the menu bar away from the area on which 
the patient should remain focused. It will be appreciated that techniques for moving a 
menu bar to various locations on a screen are known to those skilled in the art of 
designing user interfaces. 

10 FIGURE 21 is a flow chart of an exemplary routine for implementing a zoom 

feature in the aesthetic imaging program. The zoom feature allows a user to increase 
the scale of the image to better view a selected area and to improve the ability of the 
user to edit fine details in the image. At a block 400, a zoom point is selected by a 
user. The zoom point identifies the center of the image to be expanded under the 

1 5 control of the user. At a block 402, the picture is adjusted to position the zoom point 
at the center of the monitor. Centering the picture ensures that as the image is 
enlarged, the portion of the image surrounding the zoom point will be displayed. At a 
block 404, the user is allowed to input a desired magnification factor. Preferably, the 
magnification factor is selected by floating the pen from the bottom (minimum 

20 magnification) to the top (maximum magnification) of the tablet. As the user floats 
the pen over the tablet, the image on the monitor is correspondingly magnified and 
redisplayed at a block 406. With each redisplay of the image, at a block 408 a test is 
made to see if the user has frozen the image by pressing or tipping the pen. Once the 
image is frozen at a desired magnification, a user can manipulate the image using the 

25 array of drawing tools described above. It will be appreciated that for very fine work, 
such as removing small wrinkles surrounding a patient's eyes, the ability to magnify 
the image increases the quality of the editing that may be achieved. 

Several refinements of the zoom feature may be incorporated in the aesthetic 
imaging program to improve the results of the zoom. For example, a smoothing 

30 function may be incorporated in the zoom feature to ensure that as the image is 
magnified it does not become "pixelly" or grainy. The smoothing function may be 
implemented in software. Preferably, however, the smoothing function is 
implemented in hardware, such as a smoothing feature provided in the Targa 2000 
board described above and incorporated in the aesthetic imaging system. Further, 

35 feedback may be provided to the user in the form of a numerical display on the image 



WO 98/20458 



-38- 



PCT/US97/20394 * 



to indicate the approximate magnification as the user floats the pen from the bottom 
to the top of the tablet. Other means can also be implemented to allow the user to 
select the desired magnification, including a pull-down menu or numerical entry. 

Yet another view option provided to the user in the aesthetic imaging system 
5 is an Emboss option. It has been found that the emboss viewing option is very helpful 
in allowing a user to discern wrinkles or other skin imperfections in a displayed image. 
By selecting the emboss option, the user causes an image of the patient to be 
displayed in a gray scale. The emboss option displays an image that is similar to an 
etching made of a three-dimensional raised surface. A two dimensional image is 

10 portrayed, with the depth of the raised surface indicated by a darker shade of gray. 
The emboss option removes any deceptive information conveyed by the color or 
shading of the skin of the patient and allows any raised or depressed areas to be 
clearly highlighted. When viewing wrinkles or other imperfections on a patient, the 
emboss option therefore clearly identifies the raised or depressed features over the 

1 5 smooth skin of the patient. 

Preferably, the emboss option is implemented using a function provided in the 
Targa 2000 board described above and incorporated in the aesthetic imaging system. 
The particular function used to manipulate a bitmap contained in the Targa board 
buffer is: 

20 setResizerToEmbossfconst BlttListSize dsize, int dx, mi dy, int scale=0xl00) 

where: 

dx and dy = the offset of the emboss operation; and 
scale = a "pressure" of the emboss. 

After selecting the emboss option, the user may manipulate the offset of the 
25 emboss by floating the pen over the tablet. Floating the pen parallel to the x-axis of 
the tablet adjusts the dx parameter in the emboss function, and floating the pen parallel 
to the y-axis adjusts the dy parameter in the emboss function. It has been found that 
various features on the skin of a patient can be brought into greater clarity by 
adjusting the offsets of the embossed image. A user may therefore focus on the 
30 desired feature that they wish to show a patient. When the desired offsets are 
selected, the user may freeze the offsets by tipping the pen on the tablet. A user may 
then adjust the scale parameter by floating the pen parallel to the x-axis of the tablet. 
Again, it has been found that various skin features can be brought into greater clarity 
by adjusting the pressure of the emboss. When the desired pressure results are 
35 achieved, the user may freeze the image by tipping the pen on the tablet. 
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VI. Post Editing Image Manipulation 

The aesthetic imaging system 20 contains several modules to allow an image 
to be manipulated to improve the quality of the image, to analyze the image, or to 
prepare the image for meetings or presentations. FIGURES 22A-22D illustrate an 
5 exemplary routine for implementing some of the modules in the aesthetic imaging 
system. The modules are accessible from a pull-down menu or other function key. 

The first module that the user may select is a color correction module 500 that 
allows the user to adjust the color of an image so that the image matches the color of 
a target image. If the user has selected the color correction module, at a block 502 

10 the user is allowed to select an original image and a target image. FIGURE 23 depicts 
a preferred screen of the aesthetic imaging system when the color of an image is to be 
corrected. On the left hand side of the screen, a number of stored images 620a, 620b, 
620c, 620d, and 620e are displayed in an image index 620. From the image index, the 
user may select an original image to be manipulated and a target image from which the 

15 color is to be copied. The user selects the original and target images by moving the 
cursor over the desired image, selecting the image by tipping the pen on the tablet, 
and dragging the image to the appropriate location on the screen. The original image 
is selected by dragging an image from the image index 620 to an original image 
box 622, while the target image is selected by dragging an image to a target image 

20 box 624. When selected, each of the images is displayed in the appropriate box by the 
aesthetic imaging system. 

Returning to FIGURE 22A, at a block 504 the user is allowed to select 
comparison points on the original image and the target image. An original 
comparison point 626 is selected on the original image by moving the cursor to the 

25 appropriate point on the original image and tipping the pen on the tablet. Similarly, a 
target comparison point 628 is selected on the target image by appropriately moving 
the cursor and tipping the pen on the tablet. As discussed below, the comparison 
points define the color correction that is made to the original image. 

The image captured by the image capture board in the aesthetic imaging 

30 system is comprised of a number of pixels, each pixel having an 8-bit red component, 
an 8-bit green component, and an 8-bit blue component. At a block 506, a 
comparison is made of the red, green, and blue (RGB) components of the pixels in the 
regions immediately surrounding the selected comparison points. That is, the RGB 
components of the pixels in the region immediately surrounding the original 

35 comparison point are compared with the RGB components of the pixels in the region 
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immediately surrounding the target comparison point. The purpose of the comparison 
is to allow the color of the original image to be modified to closely approximate the 
color of the target image. At a block 508, an offset of the difference in color between 
the original image and the target image is calculated. The offset is calculated by 
5 subtracting the average of the red, green, and blue components of the pixels at the 
original comparison point from the average of the red, green, and blue components of 
the pixels, respectively, at the target comparison point. Preferably, the RGB 
components in a 9 x 9 pixel region surrounding each comparison point are averaged 
to arrive at an approximate value for the red component, the green component, and 

10 the blue component in each selected region. 

After the offset is calculated, the image in the original image box 622 is 
modified by adding the calculated offset to the red component, the green component, 
and the blue component of each pixel in the image. For example, if the average RGB 
components of the target image are less than the average RGB components of the 

15 original image at the comparison point, the calculated offset will be negative, thereby 
subtracting from the RGB components of each pixel in the original image. 
Conversely, if the RGB components in the target image are greater than the RGB 
components in the original image, the offset will be positive, thereby adding to the 
RGB components of each pixel in the original image. 

20 At a block 512, the modified original image is displayed to the user as an 

adjusted image in an adjusted image box 630. The result of the modification made to 
the original image is to change the overall color of the image. If the target 
comparison point contains more blue than the original comparison point, the original 
image will be bluer when displayed in the adjusted image box 630. Conversely, if the 

25 target comparison point contains more red than the original comparison point, the 
original image will be displayed as a redder image in the adjusted image box 630. 

The disclosed technique for correcting the color of an image allows a user to 
quickly match the color of two images that were taken under different conditions so 
that the images may be accurately compared. For example, images may be captured 

30 by the aesthetic imaging system using a digital camera under different lighting 
conditions, or with different aperture settings. Moreover, images that are scanned 
into the aesthetic imaging system from photographs will likely have a different color 
composition than images that are captured from a digital camera. The variations in 
color make it difficult to compare two images accurately. The disclosed color 

35 correction module allows a user to quickly select a target image that has a desired 
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color composition, and modify an original image to match the target image so that an 
accurate comparison may be made between the images. 

At a decision block 5 14, the user may continue to adjust the color of the image 
if the desired color is not achieved. At a block 504, the user may move the original 
5 comparison point 626 or the target comparison point 628 to change the image 
depicted in the adjusted image box 630. Such modifications occur in real time, 
allowing the user to quickly try a variety of different color corrections. If the desired 
color of the image is achieved, at a block 516 the adjusted image contained in the 
adjusted image box 630 is stored. It will be appreciated that the adjusted image may 

10 be stored over the old original image, or stored at a new memory location to preserve 
both the original image and the adjusted image. Once an offset is calculated, the user 
may also continue to modify a number of images using the same offset. To continue 
to modify images, the user selects another image from the image index 620 and drags 
the image to the original image box 622. The selected original image is automatically 

15 modified using the current offset, and displayed in the adjusted image box 630. The 
adjusted image is then stored. In this manner, a user can quickly adjust the color of a 
number of images so that the images may be accurately compared. 

Returning to FIGURE 22A, the second module that the user may select is an 
orientation correction module 520 to allow the size and orientation of an image to be 

20 corrected. If the user has selected the orientation correction module, at a block 522 
the user is allowed to select an original image and a target image from an image index. 
As depicted in FIGURE 24, an original image is selected from the image index 620 
and displayed in an original image box 640. A target image is selected and displayed 
in a target image box 642. Oftentimes, the orientation or size of the patient's image in 

25 the original image will be different from the target image. For example, in the 
representative images shown in FIGURE 24, the patient's head in the original image is 
tilted slightly from the orientation in the target image. To correct for the patient tilt in 
the original image, at a block 524 the user selects at least three reference points 644 
on the target image. At a block 526, the user selects at least three corresponding 

30 reference points 648 on the original image. The reference points selected on the 
original image should be located at approximately the same location on the patient's 
image as they are located on the target image. 

At a block 528, a calculation is made to determine the transform necessary to 
rotate and size the original image so that it will be the same size and orientation as the 

35 target image. Those skilled in the art will appreciate that the relative size of the 
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original and the target images may be determined from the distance between the 
reference points on each of the images. Moreover, the orientation of the two images 
may be determined by the relative location of the two sets of reference points on the 
screen. Based on a comparison of the reference points 648 on the original image and 
5 the reference points 644 on the target image, a transform is constructed that will 
rotate and scale the original image to arrive at the size and orientation of the target 
image. While preferably three reference points are used to arrive at the 
transformation, it will be appreciated that additional reference points could be added 
to the original and target images to improve the accuracy of the transform. 

10 At a block 530, the original image is rotated and sized in accordance with the 

transform to match the size and orientation of the target image. At a block 532, the 
modified original image is displayed as an adjusted image in the adjusted image 
box 644. As shown in FIGURE 24, if the data from the original image is bounded by 
the screen size, that is, the number of pixels in the patient image that is stored in the 

15 aesthetic imaging system corresponds to the number of pixels displayed on the screen, 
portions of the image may be lacking when the image is rotated or sized. For 
example, the image of the patient in FIGURE 24 is missing a portion 646 of the head 
and a portion 649 of the neck. Those skilled in the art will appreciate that several 
techniques could be used to account for the missing portions. For example, the 

20 missing portions may be displayed in a background color or interpolated to complete 
the image. Alternatively, the size of the target image and the adjusted image may be 
varied so that the adjusted image completely fills the display without missing portions. 

At a decision block 534, the user may continue adjusting the orientation and 
size of the image if the desired orientation and size is not achieved. To continue to 

25 adjust the size and orientation of the original image, the user moves any of the 
reference points on the target image or the original image at a block 524. Moving the 
reference points causes the adjusted image depicted in the adjusted image box 644 to 
be redisplayed in real time. If the user is satisfied with the adjusted image, at a 
block 536 the adjusted image is stored. The adjusted image may be stored to replace 

30 the original image, or may be stored as an additional image for later manipulation. 

With reference to FIGURES 22B-22C, the third module that the user may 
select is a measurements module 540 to perform measurements of an image. Four 
types of measurements may be made: an angle measurement, an area measurement, a 
distance measurement, or a proportion measurement. Each measurement option 

35 allows a different measurement to be displayed on the image without changing the 
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underlying image. While each measurement option will be discussed as being 
implemented on a separate image below, it will be appreciated that the various 
different measurements may be combined on a single image. 

At a decision block 542, a test is made to determine if the user has selected the 
5 angle measurement option from the measurement module. FIGURE 25 A depicts the 
screen of the aesthetic imaging system when the user selects the angle measurement 
option. At a block 544, the user initially selects an origin 650 for the angle 
measurement by moving a cursor 656 to the appropriate location on the screen and 
tipping the pen on the tablet. To make an angle measurement, at blocks 546 and 548 

10 the user must define a first line 658 and a second line 660 that intersect at the origin. 
To define the first line, a dashed line 652 is displayed that passes through the origin 
after the origin has been selected. A solid line segment 654 extends from the origin 
along the dashed line to a point where the cursor 656 is located. Movement of the 
cursor 656 along the dashed line 652 causes the length of the solid line segment 654 

15 to change. Movement of the cursor around the origin 650 causes the angle of the 
dashed line with respect to the origin to change. When a desired location of the 
dashed line is achieved, such as parallel with a patient's feature, the user tips the pen 
to set the location and length of the first line 658. After the first line is defined, the 
dashed line 652 extending through the origin is again displayed to allow the user to 

20 define the second line. Movement of the cursor changes the direction and length of 
the line segment along the dashed line. As depicted in FIGURE 25 A, the second line 
660 is defined so that it forms an angle 662 with respect to the first line 658. The user 
selects the orientation and length of the second line by tipping the pen when the cursor 
is at the desired location. 

25 After the user has defined the two lines on the patient's image, at a block 550 

the angle between the two lines is calculated and displayed on the image. Two lines 
that intersect at the origin 650 will define two angles, one greater than or equal to 
180 degrees, and one less than or equal to 180 degrees. Preferably, the aesthetic 
imaging system automatically displays the calculated angle that is less than or equal to 

30 180 degrees. The angle is displayed at a location 664 that is opposite the defined 
angle. While preferably the angle is displayed in degrees, it will be appreciated that 
the angle may also be displayed in other units of measure, such as radians. 

In addition to displaying a single angle, the aesthetic imaging system also 
allows the user to link multiple angles together. For example, as shown in 

35 FIGURE 25 A, three angles are defined by a common line 666. To allow the user to 
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chain multiple angles together, at a decision block 552 a test is made to determine if 
the user has enabled a chaining option. If the user has enabled the chaining option, at 
a block 554 the user is allowed to define additional lines following the definition of the 
first two lines. That is, after each line is defined, a dashed line is redisplayed from the 
5 end of the previously defined line to allow the user to define an additional line having 
a desired length and direction that starts from the end of the previously defined line. 
The routine then returns to block 550 where the angle between the last two defined 
lines is calculated and displayed. Each additional line added by a user therefore 
defines an additional angle. As depicted in FIGURE 25 A, the chaining option allows 

10 the user to outline portions of the image to quickly and accurately obtain angle 
measurements around a patient's feature. 

Returning to FIGURE 22B, if the angle measurement option is not selected, a 
test is made to determine if the user has selected the area measurement option at a 
decision block 558. If the user has selected the area measurement option, the user 

15 must first calibrate the image at a block 560. Calibration refers to the process of 
assigning the image a scale so that measurements made on the image would equal 
equivalent measurements on the actual patient. FIGURE 25B depicts the image 
during the calibration process. The user selects a first calibration point 680 and a 
second calibration point 682 by moving the cursor to an appropriate location and 

20 tipping the pen. To visually indicate the calibration distance to be entered, a line 684 
is shown connecting the two calibration points. When the user has selected the two 
calibration points, a dialogue box 686 is displayed to allow the user to enter a known 
distance between the two calibration points. In order to accurately calibrate the image 
the user must measure the (two-dimensional) distance on the actual patient between 

25 the two calibration points. When the actual measurement is known, the user enters 
the distance into the dialogue box 686 and the dialogue box and line 684 are removed 
from the image. Each image must only be calibrated a single time. Once the user has 
calibrated the image, accurate distance or area measurements may be made of the 
image. 

30 The area measurement option allows the user to calculate and display the area 

of features on the patient's image. After calibrating the image, at a block 562 the user 
defines an area 688 to be measured on the image by circling the area. For example, 
the user has circled the lips on the image of the patient shown in FIGURE 25B. When 
the area has been defined, the area is distinguished from the remainder of the image by 

35 filling the area with cross-hatching 690. At a block 564, the size of the area defined 
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by the user is calculated by known methods and displayed at a location 690 adjacent 
the defined area. Those skilled in the art will appreciate that the calculation of the 
area 690 is a simple matter once the image has been calibrated. The area may be 
expressed in English units, metric units, or any other units of measure used to 
5 calibrate the image. Further, the user may define any area within the image to be 
measured. 

Similar to making an area measurement, the user may also make a distance 
measurement on the image. At a decision block 568, a test is made to determine if the 
user has selected the distance measurement option. FIGURE 25C depicts an image in 

10 which a user has made several distance measurements. If the user has not calibrated 
the image, at a block 570 the image must first be calibrated in the manner described 
above. If the image is already calibrated, at a block 572 the user defines a line 
segment by selecting a first end point 700 and a second end point 702. After selecting 
the two end points, a line 704 is displayed on the image connecting the two end 

15 points. At a block 574, the two-dimensional distance between the two points is 
calculated and displayed at a location 706 adjacent the line. Those skilled in the art 
will appreciate that calculating a two-dimensional distance on the image is 
straightforward once the image has been calibrated. The distance may be displayed in 
English units, metric units, or other standard units of measure. 

20 As with the angle measurements, the aesthetic imaging system also allows a 

user to chain multiple distance measurements together. If the user has selected the 
chaining option at a decision block 576, the user is allowed to enter another end point 
after the first two end points are displayed at a block 578. The third and subsequent 
end points are connected to the preceding end point by a line, and the distance of each 

25 line displayed. As depicted in FIGURE 25C, the user may use the chaining option to 
quickly enter a number of lines 708 to measure around a desired area or between 
multiple points. 

With reference to FIGURE 22C, the fourth measurement that a user may 
make on an image is a proportion measurement. At a decision block 580, a test is 

30 made to determine if the user has selected the proportion measurement option. If the 
user has selected a proportion measurement, the user is presented the option of 
making a horizontal proportion measurement (comparing the horizontal relationship 
of features in the image) or a vertical proportion measurement (comparing the vertical 
relationship of features in the image). At a decision block 582 a test is made to 

35 determine if the horizontal or the vertical proportion measurement has been selected. 



WO 98/20458 



-46- 



PCIYUS97/20394 % 



If a vertical proportion measurement is selected, at a block 584 the user defines a 
number of horizontal lines on an image of the patient. As depicted in FIGURE 25D, a 
number of lines 720a, 720b, . . . 720e that are defined by the user split the image into 
a number of regions 722a, 722b, 722c, and 722d. The user preferably defines the lines 
5 by floating the pen over the tablet and tipping the pen at the appropriate locations as 
the cursor moves from top to bottom on the image. While the lines are shown in 
FIGURE 25D as extending across the entire image, it will be appreciated that the end 
points of the lines may be adjustable to allow the length of the lines to correspond to a 
desired contour, such as the exterior of the patient's face. 

10 At a block 586, the distance between the first and last horizontal lines defined 

by the user is determined. At a block 588, the total distance is used to determine the 
percentage that each region encompasses of the total distance between the starting 
and the ending lines. For example, as shown in FIGURE 25D, the distance between 
lines 720d and 720e is divided by the distance between the first line 720a and the last 

15 line 720e to arrive at the percentage associated with region 722d. The percentage 
corresponding to each region is then displayed at a location 724 within each region. 
For example, the region 722 encompasses 27% of the total distance between 
lines 720a and 720e. Dividing an image to determine how the image is proportioned 
is extremely advantageous in the aesthetic imaging industry since certain proportions 

20 are considered to be more desirable than others. 

FIGURE 25D depicts an image divided by horizontal lines to determine the 
vertical proportions of the patient. Alternatively, an image may be divided by vertical 
lines to determine the horizontal proportions of the patient. If the horizontal 
proportions of the patient are to be measured, the user defines a plurality of vertical 

25 lines including a first line and a last line at a block 590. At a block 592, the total 
distance between the first line and the last line is calculated. At a block 594, the 
distance between each of the lines is compared with the total distance to arrive at a 
percentage of the total distance defined by the first and the last lines. The percentages 
are then displayed to the user within the regions defined by the vertical lines. 

30 With reference to FIGURE 22D, yet another module provided in the aesthetic 

imaging system is a label module 600 to allow labels to be added to an image. 
FIGURE 26 portrays a representative screen when the user has selected the labeling 
module. At a block 602, the user defines a line segment on an image by selecting a 
first end point and a second end point. For example, as depicted in FIGURE 26, a 

35 user may define a first end point 730 near the eyebrow on a patients image, and a 
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second end point 732 located at a distance from the eyebrow. After defining the line 
segment, at a block 604 the user enters a label corresponding to the line segment. As 
shown in FIGURE 26, a transparent box 734 is displayed to the user at the location 
where the label will be displayed The user enters the label using a keyboard or other 
5 technique, such as selecting predefined labels from a pull-down menu. The box 734 
used to enter the label is preferably transparent to allow the user to see the image of 
the patient through the box when a label is being entered. 

At a decision block 606 the user is allowed to enter multiple labels, each of 
which are linked to one or more structures on the drawing. For example, a label 736 

10 may be linked to two or more line segments that point to different structures on the 
image. In FIGURE 26, the label "Lips" is connected by two line segments to each of 
the patient's lips. The labels are also linked to the line segments so that if the labels 
are moved, the line segments are correspondingly moved to continue to connect the 
label with the associated structure. For example, the label "Eye," which points to the 

15 eye of the image in FIGURE 26, may be selected by the user and dragged from a first 
location 738 to a second location 740 as indicated by the directional arrow. Moving 
the label "Eye" automatically moves the line segment associated with the label so that 
the label and the structure remain linked. 

The aesthetic imaging system modules disclosed above are particularly useful 

20 for allowing an imaging professional to manipulate an image to improve the 
presentation of the image to the patient or to colleagues. Such modules allow a more 
accurate comparison to be made between two or more images. The result is a more 
satisfying and beneficial imaging session for a patient. 

VTI Treatment Simulations 

25 The aesthetic imaging system of the present invention incorporates a module 

for simulating the results of laser resurfacing treatment. FIGURE 27A depicts a 
representative screen of the aesthetic imaging system when the laser resurfacing 
module has been selected. The laser resurfacing module allows the aesthetic imaging 
system to display an image of the patient that approximates how the patient will 

30 appear immediately following a laser resurfacing treatment and throughout the healing 
process. 

The steps required to simulate the results of the laser resurfacing treatment are 
depicted in the flow chart of FIGURE 27B. At a block 754, the user initially selects 
an image to be edited, and defines a region in which the laser resurfacing treatment is 
35 to be simulated. The region is defined by the user by circling the desired area. For 
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example, in FIGURE 27A, the user has selected a patient image 750 and circled a 
region 752 that encompasses the majority of the cheek, chin, and jaw of the patient. 

The unedited image is stored in a buffer by the aesthetic imaging system as an 
original image 762, as depicted diagrammatically in FIGURE 27C. The aesthetic 
5 imaging system also uses the region 752 that is circled by the user to create an alpha 
mask 764. The alpha mask is constructed using an alpha key channel associated with 
the patient image captured by the image capture board. In addition to the 24-bit RGB 
pixel values associated with an image, each image includes an alpha key channel that 
provides information about the opacity of each pixel. For each pixel in the image, the 

10 alpha key channel contains an 8-bit value that represents the transparency of the pixel 
when displayed or superimposed over another pixel. The alpha mask 764 created by 
the aesthetic imaging system uses the alpha key channel to make a mask for 
manipulating the original image. The mask corresponds to the shape of the defined 
region 752. That is, the alpha channel values corresponding to pixels outside the 

15 region 752 are set to zero (transparent) in the alpha mask, and the majority of the 
alpha channel values corresponding to pixels within the region 752 are set to 255 
(opaque) in the alpha mask. Preferably, the edges of the mask are gradiated to 
smooth the boundary of the mask. FIGURE 27E depicts a representative alpha 
mask 764 having a gradiated boundary. Areas outside the mask contain an alpha 

20 value of zero, and areas at the center of the mask contain an alpha value of 255. 
Around the boundary of the mask, the alpha values incrementally increase from 0 to 
255. While three gradiations (of 50, 125, and 200) are depicted in FIGURE 27E, it 
will be appreciated that a greater or lesser number of gradiations may be included in 
the mask. The alpha mask is stored after creation by the aesthetic imaging system. 

25 It has been found that slightly blurring an image produces an accurate 

approximation of the results that laser resurfacing treatment can achieve. At a 
block 756, the selected region 752 of the image is therefore blurred to simulate the 
treatment results. FIGURE 27D diagrammatically depicts the technique for blurring 
the portion of the original image 762 that is associated with the region 752. The 

30 original image 766 is initially filtered by a low pass filter 766 to produce a blurred 
image 770. The alpha mask 764 is then added to the blurred image 770 and 
superimposed over the original image 762 so that only the portion of the blurred 
image corresponding to the defined region 752 is displayed to the user. The amount 
of blurring depicted in the image is varied by changing the opacity of the alpha 

35 mask 764. As the mask becomes more transparent, less blurring of the image occurs. 
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As the mask becomes more opaque, more blurring of the image occurs. Preferably, 
the opacity of the alpha mask 764 is tied to the movement of the cursor to allow the 
user to change the blurring of the image by floating the pen from the top to the 
bottom of the tablet. 

5 Because of computational limitations, the blurred image 770 is preferably only 

calculated once and the amount of blurring varied by changing the opacity of the alpha 
mask 764 as described above. Alternatively, those skilled in the art will recognize that 
with additional processing power, the amount of blurring may be varied by directly 
changing the response of the low pass filter 766. Using the alternative technique, the 

10 low pass filter response may be tied to the movement of the cursor so that the user 
may directly vary the blurriness of the selected region by floating the pen from the top 
to the bottom of the tablet. 

When the user has achieved a desired amount of blurriness to simulate the 
representative treatment results within the region 752, the user freezes the image of 

15 the patient by tipping the pen on the tablet. An image 772 representative of the 
anticipated treatment results (after the patient has healed) is then stored in a buffer in 
the aesthetic imaging system. 

To simulate the redness and inflammation immediately following the 
resurfacing treatment, at a block 758 the user selects an amount of redness to be 

20 applied to the treated region. The amount of redness may be preset, or selected by the 
user from a scale of redness. FIGURE 27D diagrammatically represents the process 
for displaying the selected region 752 in a reddened state. The alpha mask 764 is 
added to a solid image 774 containing pixels of the desired redness, and the alpha 
channel values in the resulting mask changed to make the mask semi-transparent. The 

25 mask and colored image is then superimposed over the image 772 of the 
representative treatment results after healing. The result is an image 776 that is 
representative of the image of the patient after the treatment has been performed, but 
before the patient has healed. 

Returning to FIGURE 27B, the user is allowed to fade the redness of the 

30 image at a block 760 to show how the patient will likely look during the healing 
process. In order to fade the redness of the image, the alpha values in the alpha 
mask 764 are reduced until the entire mask is transparent. As the mask becomes more 
transparent, more of the underlying image is visible to the user. Preferably, the 
transparency of the alpha mask 764 is tied to the movement of the cursor to allow the 

35 user to change the redness of the image by floating the pen from the top to the bottom 
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of the tablet. It will be appreciated that the healing process is typically very fast in the 
first few days following treatment, after which the healing process becomes less 
noticeable. To accurately simulate the pace of the healing process, the transparency 
of the mask 764 may change very quickly in response to an initial cursor movement 
5 before tapering off. Ultimately, only the image 772 of the patient that is 
representative of the treatment results is displayed to the user. 

The laser resurfacing simulation module allows the aesthetic imaging system to 
display an image of a patient before laser resurfacing treatment, an image of the 
patient with redness and irritation immediately after treatment, and a simulation of the 

10 healing process leading to the anticipated results that are achievable by the treatment. 
While the discussion above reflects a series of steps performed by the user, it will be 
appreciated that the entire process could be automated by the aesthetic imaging 
system. In an automated system, the user would merely select the region 752 in which 
the treatment is to be simulated. The aesthetic imaging system would then 

15 automatically apply a preselected amount of blurriness and redness, and slowly cycle 
through the display of the healing process before ending with an image of the 
achievable results. 

While the preferred embodiment of the invention has been illustrated and 
described, it will be appreciated that various changes can be made therein without 

20 departing from the spirit and scope of the invention. For example, the advantages of 
the aesthetic imaging system are not limited to imaging for cosmetic surgery only, but 
are applicable to a number of medical imaging fields, including endoscopy, pathology, 
and hair restoration. Those skilled in the art will recognize adaptations that may be 
made to accommodate these medical imaging fields without departing from the scope 

25 of the invention. Also, a different style of pointing device may be used in lieu of the 
pen and tablet. 
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The embodiments of the invention in which an exclusive property or privilege 
is claimed are defined as follows: 

1 . In an aesthetic imaging system of the type that manipulates an image of 
a patient to simulate the effects of a medical procedure, a method for simulating the 
results of a laser resurfacing treatment, comprising the steps of: 

obtaining an original image of the patient; 
5 defining an area in the image in which the laser resurfacing is to be simulated; 

creating a mask whereby the areas outside the region in which the laser 
resurfacing is not to be simulated is transparent and the areas in which the laser 
resurfacing is to be simulated is opaque; 

creating a blurred copy of the original image in the area in which the laser 
1 0 resurfacing is to be simulated; and 

adding the mask to the blurred copy of the original image and superimposing 
the sum of the blurred image with the mask over the original image to simulate the 
laser resurfacing. 

2. The method of Claim 1, further comprising a method for connecting 
the color of an image comprising the steps of: 

selecting one or more comparison points in an original image and a target 

15 image; 

calculating an average red, green, and blue components of the pixels near the 
comparison points in the original image and the target image; 

determining a difference in the average red, green, and blue components of the 
pixels near the comparison points in the original and target images; and 
20 adjusting the red, green, and blue components of the pixels near the target 

points in the original image to equal the red, green, and blue components of the pixels 
in the target image. 
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3. The method of Claim 1, further comprising a method of connecting the 
orientation of an image, comprising the steps of: 

selecting a number of reference points in an original image and a target image; 

computing a transform that relates the reference points in the original image 

and in the target image; and 

adjusting the size and location of the original image with the transform to 

5 connect the orientation of the original image. 

4. The method of Claim 1, further comprising the steps of allowing a user 
to compute measurements on the original image. 

5. The method of Claim 1, further comprising allowing a user to add 
labels to the original image. 
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AMENDED CLAIMS 

[received by the International Bureau on 30 April 1998 (30.04.98); 
original claims 1-5 replaced by new claims 1-13 (3 pages)] 

5 1. In an aesthetic imaging system allowing the manipulation of digital 

images comprised of a plurality of pixels, a method of displaying an edited digital 
image corresponding to an original digital image wherein the edited digital image 
differs from the original digital image by at least one edit made to the original digital 
image, the aesthetic imaging system including a processor, a memory, a monitor, and 
10 means for controlling a cursor displayed on the monitor, the method comprising: 

defining a plurality of regions in said edited digital image, each of the plurality 
of regions containing at least one edit that distinguishes each of the plurality of 
regions from a corresponding region in said original digital image; 

allowing a user to select from the plurality of regions a subset of regions; and 
IS displaying a prioritized digital image containing only the edits of the original 

digital image that are present in the subset of regions. 

2. The method of Claim 1 wherein the plurality of regions are defined by 
encircling the region on the edited digital image with the means for controlling the 
cursor displayed on the monitor. 

20 3. The method of Claim 1 wherein the plurality of regions are 

automatically defined by the aesthetic imaging system. 

4. The method of Claim 1 further comprising the step of assigning a tag 
corresponding to each of the plurality of regions. 

5. The method of Claim 4 wherein the tag is a textual identifier. 

25 6. The method of Claim 5, wherein the subset of regions are selected 

from a list of tags corresponding to each region. 

7. The method of Claim 1 wherein the original image and the prioritized 
image are displayed side-by-side. 

8. The method of Claim 1 wherein the prioritized image is superimposed 
30 over the original image, an opacity of the prioritized image being selectable by a user 

to allow the user to view a composite image of the prioritized image and the original 
image, the Composite image thereby displaying a variable amount of the edits 
performed on the original image. 
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5 9. The method of Claim 8 wherein the opacity of the prioritized image 

may vary between zero and 100 percent. 

10. In an aesthetic imaging system allowing the manipulation of digital 
images comprised of a plurality of pixels, each pixel having an alpha channel value 
indicating the opacity of the pixel, wherein an alpha channel value of zero indicates a 

10 transparent pixel and an alpha channel value of 255 indicates an opaque pixel, a 
method simulating the results of a laser resurfacing treatment, comprising: 

selecting an original digital image from a memory of the aesthetic imaging 

system; 

defining a region on the original digital image; 
1 s setting the alpha channel values of the pixels outside the defined region at zero 

and setting the alpha channel values of pixels inside the defined region between zero 
and 255 to create an alpha mask with a desired opacity; 

filtering the selected digital image to produce a blurred digital image; 
adding the alpha mask to the blurred digital image to create a resulting blurred 
20 digital image; 

superimposing the resulting blurred digital image over the original digital 
image to create an edited digital image; and 
displaying the edited digital image. 

1 1 . The method of Claim 1 0 further including the step of: 

25 adding a solid red digital image to the alpha mask, wherein the degree of 

redness of the solid red digital image can be predetermined and automatically varied 
or can be varied by the user by varying the degree of redness of the solid red digital 
image or varying the opacity of the pixels of the alpha mask. 

12. In an aesthetic imaging system allowing the manipulation of digital 
30 images comprised of a plurality of pixels having red, green, and blue (RGB) 

components, a method of correcting the color of an original digital image to 
substantially match the color of a target digital image comprising: 

selecting an original digital image and a target digital image from a memory of 
the aesthetic imaging system 
35 selecting a comparison point on the original digital image and a comparison 

point on the target digital image; 
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5 calculating an offset value by subtracting an average of the RGB components 

of the pixels in a region surrounding the comparison point in the original image from 
an average of the RGB components of the pixels in a region surrounding the 
comparison point in the target image, 

adding the offset value to the RGB components of each pixel in the original 
10 image to correct the color of the original digital image; and 
displaying the color corrected original digital image. 

13. In an aesthetic imaging system allowing the manipulation of digital 
images comprised of a plurality of pads, a method of correcting the size and 
orientation of an original digital image to substantially match a target digital image 
IS comprising: 

selecting an original digital image and a target digital image from a memory of 
the aesthetic imaging systems; 

selecting at least three reference points on the original digital image and at 
least three corresponding reference points on the target digital image; 
20 calculating a degree of rotation and a size adjustment based on the three 

reference points in the original and target digital images; 

rotating the original digital image based on the calculated degree of rotation 
and scaling the original digital image based on the calculated size adjustment to create 
a corrected digital image; and 
25 displaying the corrected digital image. 
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STATEMENT UNDER ARTICLE 19(1) 
Applicant requests that the claims be amended by substituting the attached replacement claim 
sheets. Claims 1-5 have been cancelled and new claims 1-13 have been added. 
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